git/builtin/sparse-checkout.c

1049 lines
28 KiB
C
Raw Normal View History

#include "builtin.h"
#include "config.h"
#include "dir.h"
#include "environment.h"
#include "gettext.h"
#include "object-file.h"
#include "object-name.h"
#include "parse-options.h"
#include "pathspec.h"
#include "repository.h"
#include "run-command.h"
#include "strbuf.h"
#include "string-list.h"
#include "cache-tree.h"
#include "lockfile.h"
#include "resolve-undo.h"
#include "unpack-trees.h"
#include "wt-status.h"
#include "quote.h"
#include "setup.h"
#include "sparse-index.h"
sparse-checkout: set worktree-config correctly `git sparse-checkout set/init` enables worktree-specific configuration[*] by setting extensions.worktreeConfig=true, but neglects to perform the additional necessary bookkeeping of relocating `core.bare=true` and `core.worktree` from $GIT_COMMON_DIR/config to $GIT_COMMON_DIR/config.worktree, as documented in git-worktree.txt. As a result of this oversight, these settings, which are nonsensical for secondary worktrees, can cause Git commands to incorrectly consider a worktree bare (in the case of `core.bare`) or operate on the wrong worktree (in the case of `core.worktree`). Fix this problem by taking advantage of the recently-added init_worktree_config() which enables `extensions.worktreeConfig` and takes care of necessary bookkeeping. While at it, for backward-compatibility reasons, also stop upgrading the repository format to "1" since doing so is (unintentionally) not required to take advantage of `extensions.worktreeConfig`, as explained by 11664196ac ("Revert "check_repository_format_gently(): refuse extensions for old repositories"", 2020-07-15). [*] The main reason to use worktree-specific config for the sparse-checkout builtin was to avoid enabling sparse-checkout patterns in one and causing a loss of files in another. If a worktree does not have a sparse-checkout patterns file, then the sparse-checkout logic will not kick in on that worktree. Reported-by: Sean Allred <allred.sean@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-07 21:33:01 +00:00
#include "worktree.h"
static const char *empty_base = "";
static char const * const builtin_sparse_checkout_usage[] = {
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
N_("git sparse-checkout (init | list | set | add | reapply | disable | check-rules) [<options>]"),
NULL
};
static void write_patterns_to_file(FILE *fp, struct pattern_list *pl)
{
int i;
for (i = 0; i < pl->nr; i++) {
struct path_pattern *p = pl->patterns[i];
if (p->flags & PATTERN_FLAG_NEGATIVE)
fprintf(fp, "!");
fprintf(fp, "%s", p->pattern);
if (p->flags & PATTERN_FLAG_MUSTBEDIR)
fprintf(fp, "/");
fprintf(fp, "\n");
}
}
static char const * const builtin_sparse_checkout_list_usage[] = {
"git sparse-checkout list",
NULL
};
static int sparse_checkout_list(int argc, const char **argv, const char *prefix)
{
static struct option builtin_sparse_checkout_list_options[] = {
OPT_END(),
};
struct pattern_list pl;
char *sparse_filename;
int res;
setup_work_tree();
if (!core_apply_sparse_checkout)
die(_("this worktree is not sparse"));
pass subcommand "prefix" arguments to parse_options() Recent commits such as bf0a6b65fc (builtin/multi-pack-index.c: let parse-options parse subcommands, 2022-08-19) converted a few functions to match our usual argc/argv/prefix conventions, but the prefix argument remains unused. However, there is a good use for it: they should pass it to their own parse_options() functions, where it may be used to adjust the value of any filename options. In all but one of these functions, there's no behavior change, since they don't use OPT_FILENAME. But this is an actual fix for one option, which you can see by modifying the test suite like so: diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 4fe57414c1..d0974d4371 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -186,7 +186,11 @@ test_expect_success 'writing a bitmap with --refs-snapshot' ' # Then again, but with a refs snapshot which only sees # refs/tags/one. - git multi-pack-index write --bitmap --refs-snapshot=snapshot && + ( + mkdir subdir && + cd subdir && + git multi-pack-index write --bitmap --refs-snapshot=../snapshot + ) && test_path_is_file $midx && test_path_is_file $midx-$(midx_checksum $objdir).bitmap && I'd emphasize that this wasn't broken by bf0a6b65fc; it has been broken all along, because the sub-function never got to see the prefix. It is that commit which is actually enabling us to fix it (and which also brought attention to the problem because it triggers -Wunused-parameter!) The other functions changed here don't use OPT_FILENAME at all. In their cases this isn't fixing anything visible, but it's following the usual pattern and future-proofing them against somebody adding new options and being surprised. I didn't include a test for the one visible case above. We don't generally test routine parse-options behavior for individual options. The challenge here was finding the problem, and now that this has been done, it's not likely to regress. Likewise, we could apply the patch above to cover it "for free" but it makes reading the rest of the test unnecessarily complicated. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-25 10:47:00 +00:00
argc = parse_options(argc, argv, prefix,
builtin_sparse_checkout_list_options,
builtin_sparse_checkout_list_usage, 0);
memset(&pl, 0, sizeof(pl));
pl.use_cone_patterns = core_sparse_checkout_cone;
sparse_filename = get_sparse_checkout_filename();
res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL, 0);
free(sparse_filename);
if (res < 0) {
warning(_("this worktree is not sparse (sparse-checkout file may not exist)"));
return 0;
}
if (pl.use_cone_patterns) {
int i;
struct pattern_entry *pe;
struct hashmap_iter iter;
struct string_list sl = STRING_LIST_INIT_DUP;
hashmap_for_each_entry(&pl.recursive_hashmap, &iter, pe, ent) {
/* pe->pattern starts with "/", skip it */
string_list_insert(&sl, pe->pattern + 1);
}
string_list_sort(&sl);
for (i = 0; i < sl.nr; i++) {
quote_c_style(sl.items[i].string, NULL, stdout, 0);
printf("\n");
}
return 0;
}
write_patterns_to_file(stdout, &pl);
clear_pattern_list(&pl);
return 0;
}
sparse-checkout: clear tracked sparse dirs When changing the scope of a sparse-checkout using cone mode, we might have some tracked directories go out of scope. The current logic removes the tracked files from within those directories, but leaves the ignored files within those directories. This is a bit unexpected to users who have given input to Git saying they don't need those directories anymore. This is something that is new to the cone mode pattern type: the user has explicitly said "I want these directories and _not_ those directories." The typical sparse-checkout patterns more generally apply to "I want files with with these patterns" so it is natural to leave ignored files as they are. This focus on directories in cone mode provides us an opportunity to change the behavior. Leaving these ignored files in the sparse directories makes it impossible to gain performance benefits in the sparse index. When we track into these directories, we need to know if the files are ignored or not, which might depend on the _tracked_ .gitignore file(s) within the sparse directory. This depends on the indexed version of the file, so the sparse directory must be expanded. We must take special care to look for untracked, non-ignored files in these directories before deleting them. We do not want to delete any meaningful work that the users were doing in those directories and perhaps forgot to add and commit before switching sparse-checkout definitions. Since those untracked files might be code files that generated ignored build output, also do not delete any ignored files from these directories in that case. The users can recover their state by resetting their sparse-checkout definition to include that directory and continue. Alternatively, they can see the warning that is presented and delete the directory themselves to regain the performance they expect. By deleting the sparse directories when changing scope (or running 'git sparse-checkout reapply') we regain these performance benefits as if the repository was in a clean state. Since these ignored files are frequently build output or helper files from IDEs, the users should not need the files now that the tracked files are removed. If the tracked files reappear, then they will have newer timestamps than the build artifacts, so the artifacts will need to be regenerated anyway. Use the sparse-index as a data structure in order to find the sparse directories that can be safely deleted. Re-expand the index to a full one if it was full before. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-08 01:42:33 +00:00
static void clean_tracked_sparse_directories(struct repository *r)
{
int i, was_full = 0;
struct strbuf path = STRBUF_INIT;
size_t pathlen;
struct string_list_item *item;
struct string_list sparse_dirs = STRING_LIST_INIT_DUP;
/*
* If we are not using cone mode patterns, then we cannot
* delete directories outside of the sparse cone.
*/
if (!r || !r->index || !r->worktree)
return;
if (init_sparse_checkout_patterns(r->index) ||
!r->index->sparse_checkout_patterns->use_cone_patterns)
return;
/*
* Use the sparse index as a data structure to assist finding
* directories that are safe to delete. This conversion to a
* sparse index will not delete directories that contain
* conflicted entries or submodules.
*/
if (r->index->sparse_index == INDEX_EXPANDED) {
sparse-checkout: clear tracked sparse dirs When changing the scope of a sparse-checkout using cone mode, we might have some tracked directories go out of scope. The current logic removes the tracked files from within those directories, but leaves the ignored files within those directories. This is a bit unexpected to users who have given input to Git saying they don't need those directories anymore. This is something that is new to the cone mode pattern type: the user has explicitly said "I want these directories and _not_ those directories." The typical sparse-checkout patterns more generally apply to "I want files with with these patterns" so it is natural to leave ignored files as they are. This focus on directories in cone mode provides us an opportunity to change the behavior. Leaving these ignored files in the sparse directories makes it impossible to gain performance benefits in the sparse index. When we track into these directories, we need to know if the files are ignored or not, which might depend on the _tracked_ .gitignore file(s) within the sparse directory. This depends on the indexed version of the file, so the sparse directory must be expanded. We must take special care to look for untracked, non-ignored files in these directories before deleting them. We do not want to delete any meaningful work that the users were doing in those directories and perhaps forgot to add and commit before switching sparse-checkout definitions. Since those untracked files might be code files that generated ignored build output, also do not delete any ignored files from these directories in that case. The users can recover their state by resetting their sparse-checkout definition to include that directory and continue. Alternatively, they can see the warning that is presented and delete the directory themselves to regain the performance they expect. By deleting the sparse directories when changing scope (or running 'git sparse-checkout reapply') we regain these performance benefits as if the repository was in a clean state. Since these ignored files are frequently build output or helper files from IDEs, the users should not need the files now that the tracked files are removed. If the tracked files reappear, then they will have newer timestamps than the build artifacts, so the artifacts will need to be regenerated anyway. Use the sparse-index as a data structure in order to find the sparse directories that can be safely deleted. Re-expand the index to a full one if it was full before. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-08 01:42:33 +00:00
/*
* If something, such as a merge conflict or other concern,
* prevents us from converting to a sparse index, then do
* not try deleting files.
*/
if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))
return;
was_full = 1;
}
strbuf_addstr(&path, r->worktree);
strbuf_complete(&path, '/');
pathlen = path.len;
/*
* Collect directories that have gone out of scope but also
* exist on disk, so there is some work to be done. We need to
* store the entries in a list before exploring, since that might
* expand the sparse-index again.
*/
for (i = 0; i < r->index->cache_nr; i++) {
struct cache_entry *ce = r->index->cache[i];
if (S_ISSPARSEDIR(ce->ce_mode) &&
repo_file_exists(r, ce->name))
string_list_append(&sparse_dirs, ce->name);
}
for_each_string_list_item(item, &sparse_dirs) {
struct dir_struct dir = DIR_INIT;
struct pathspec p = { 0 };
struct strvec s = STRVEC_INIT;
strbuf_setlen(&path, pathlen);
strbuf_addstr(&path, item->string);
dir.flags |= DIR_SHOW_IGNORED_TOO;
setup_standard_excludes(&dir);
strvec_push(&s, path.buf);
parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);
fill_directory(&dir, r->index, &p);
if (dir.nr) {
warning(_("directory '%s' contains untracked files,"
" but is not in the sparse-checkout cone"),
item->string);
} else if (remove_dir_recursively(&path, 0)) {
/*
* Removal is "best effort". If something blocks
* the deletion, then continue with a warning.
*/
warning(_("failed to remove directory '%s'"),
item->string);
}
strvec_clear(&s);
clear_pathspec(&p);
sparse-checkout: clear tracked sparse dirs When changing the scope of a sparse-checkout using cone mode, we might have some tracked directories go out of scope. The current logic removes the tracked files from within those directories, but leaves the ignored files within those directories. This is a bit unexpected to users who have given input to Git saying they don't need those directories anymore. This is something that is new to the cone mode pattern type: the user has explicitly said "I want these directories and _not_ those directories." The typical sparse-checkout patterns more generally apply to "I want files with with these patterns" so it is natural to leave ignored files as they are. This focus on directories in cone mode provides us an opportunity to change the behavior. Leaving these ignored files in the sparse directories makes it impossible to gain performance benefits in the sparse index. When we track into these directories, we need to know if the files are ignored or not, which might depend on the _tracked_ .gitignore file(s) within the sparse directory. This depends on the indexed version of the file, so the sparse directory must be expanded. We must take special care to look for untracked, non-ignored files in these directories before deleting them. We do not want to delete any meaningful work that the users were doing in those directories and perhaps forgot to add and commit before switching sparse-checkout definitions. Since those untracked files might be code files that generated ignored build output, also do not delete any ignored files from these directories in that case. The users can recover their state by resetting their sparse-checkout definition to include that directory and continue. Alternatively, they can see the warning that is presented and delete the directory themselves to regain the performance they expect. By deleting the sparse directories when changing scope (or running 'git sparse-checkout reapply') we regain these performance benefits as if the repository was in a clean state. Since these ignored files are frequently build output or helper files from IDEs, the users should not need the files now that the tracked files are removed. If the tracked files reappear, then they will have newer timestamps than the build artifacts, so the artifacts will need to be regenerated anyway. Use the sparse-index as a data structure in order to find the sparse directories that can be safely deleted. Re-expand the index to a full one if it was full before. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-08 01:42:33 +00:00
dir_clear(&dir);
}
string_list_clear(&sparse_dirs, 0);
strbuf_release(&path);
if (was_full)
ensure_full_index(r->index);
}
static int update_working_directory(struct pattern_list *pl)
{
enum update_sparsity_result result;
struct unpack_trees_options o;
struct lock_file lock_file = LOCK_INIT;
struct repository *r = the_repository;
sparse-checkout: avoid staging deletions of all files sparse-checkout's purpose is to update the working tree to have it reflect a subset of the tracked files. As such, it shouldn't be switching branches, making commits, downloading or uploading data, or staging or unstaging changes. Other than updating the worktree, the only thing sparse-checkout should touch is the SKIP_WORKTREE bit of the index. In particular, this sets up a nice invariant: running sparse-checkout will never change the status of any file in `git status` (reflecting the fact that we only set the SKIP_WORKTREE bit if the file is safe to delete, i.e. if the file is unmodified). Traditionally, we did a _really_ bad job with this goal. The predecessor to sparse-checkout involved manual editing of .git/info/sparse-checkout and running `git read-tree -mu HEAD`. That command would stage and unstage changes and overwrite dirty changes in the working tree. The initial implementation of the sparse-checkout command was no better; it simply invoked `git read-tree -mu HEAD` as a subprocess and had the same caveats, though this issue came up repeatedly in review comments and workarounds for the problems were put in place before the feature was merged[1, 2, 3, 4, 5, 6; especially see 4 & 6]. [1] https://lore.kernel.org/git/CABPp-BFT9A5n=_bx5LsjCvbogqwSjiwgr5amcjgbU1iAk4KLJg@mail.gmail.com/ [2] https://lore.kernel.org/git/CABPp-BEmwSwg4tgJg6nVG8a3Hpn_g-=ZjApZF4EiJO+qVgu4uw@mail.gmail.com/ [3] https://lore.kernel.org/git/CABPp-BFV7TA0qwZCQpHCqx9N+JifyRyuBQ-pZ_oGfe-NOgyh7A@mail.gmail.com/ [4] https://lore.kernel.org/git/CABPp-BHYCCD+Vx5fq35jH82eHc1-P53Lz_aGNpHJNcx9kg2K-A@mail.gmail.com/ [5] https://lore.kernel.org/git/CABPp-BF+JWYZfDqp2Tn4AEKVp4b0YMA=Mbz4Nz62D-gGgiduYQ@mail.gmail.com/ [6] https://lore.kernel.org/git/20191121163706.GV23183@szeder.dev/ However, these workarounds, in addition to disabling the feature in a number of important cases, also missed one special case. I'll get back to it later. In the 2.27.0 cycle, the disabling of the feature was lifted by finally replacing the internal equivalent of `git read-tree -mu HEAD` with something that did what we wanted: the new update_sparsity() function in unpack-trees.c that only ever updates SKIP_WORKTREE bits in the index and updates the working tree to match. This new function handles all the cases that were problematic for the old implementation, except that it breaks the same special case that avoided the workarounds of the old implementation, but broke it in a different way. So...that brings us to the special case: a git clone performed with --no-checkout. As per the meaning of the flag, --no-checkout does not check out any branch, with the implication that you aren't on one and need to switch to one after the clone. Implementationally, HEAD is still set (so in some sense you are partially on a branch), but * the index is "unborn" (non-existent) * there are no files in the working tree (other than .git/) * the next time git switch (or git checkout) is run it will run unpack_trees with `initial_checkout` flag set to true. It is not until you run, e.g. `git switch <somebranch>` that the index will be written and files in the working tree populated. With this special --no-checkout case, the traditional `read-tree -mu HEAD` behavior would have done the equivalent of acting like checkout -- switch to the default branch (HEAD), write out an index that matches HEAD, and update the working tree to match. This special case slipped through the avoid-making-changes checks in the original sparse-checkout command and thus continued there. After update_sparsity() was introduced and used (see commit f56f31af03 ("sparse-checkout: use new update_sparsity() function", 2020-03-27)), the behavior for the --no-checkout case changed: Due to git's auto-vivification of an empty in-memory index (see do_read_index() and note that `must_exist` is false), and due to sparse-checkout's update_working_directory() code to always write out the index after it was done, we got a new bug. That made it so that sparse-checkout would switch the repository from a clone with an "unborn" index (i.e. still needing an initial_checkout), to one that had a recorded index with no entries. Thus, instead of all the files appearing deleted in `git status` being known to git as a special artifact of not yet being on a branch, our recording of an empty index made it suddenly look to git as though it was definitely on a branch with ALL files staged for deletion! A subsequent checkout or switch then had to contend with the fact that it wasn't on an initial_checkout but had a bunch of staged deletions. Make sure that sparse-checkout changes nothing in the index other than the SKIP_WORKTREE bit; in particular, when the index is unborn we do not have any branch checked out so there is no sparsification or de-sparsification work to do. Simply return from update_working_directory() early. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-05 02:41:39 +00:00
/* If no branch has been checked out, there are no updates to make. */
if (is_index_unborn(r->index))
return UPDATE_SPARSITY_SUCCESS;
r->index->sparse_checkout_patterns = pl;
memset(&o, 0, sizeof(o));
o.verbose_update = isatty(2);
o.update = 1;
o.head_idx = -1;
o.src_index = r->index;
o.dst_index = r->index;
o.skip_sparse_checkout = 0;
setup_work_tree();
repo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);
setup_unpack_trees_porcelain(&o, "sparse-checkout");
result = update_sparsity(&o, pl);
clear_unpack_trees_porcelain(&o);
if (result == UPDATE_SPARSITY_WARNINGS)
/*
* We don't do any special handling of warnings from untracked
* files in the way or dirty entries that can't be removed.
*/
result = UPDATE_SPARSITY_SUCCESS;
if (result == UPDATE_SPARSITY_SUCCESS)
write_locked_index(r->index, &lock_file, COMMIT_LOCK);
else
rollback_lock_file(&lock_file);
sparse-checkout: clear tracked sparse dirs When changing the scope of a sparse-checkout using cone mode, we might have some tracked directories go out of scope. The current logic removes the tracked files from within those directories, but leaves the ignored files within those directories. This is a bit unexpected to users who have given input to Git saying they don't need those directories anymore. This is something that is new to the cone mode pattern type: the user has explicitly said "I want these directories and _not_ those directories." The typical sparse-checkout patterns more generally apply to "I want files with with these patterns" so it is natural to leave ignored files as they are. This focus on directories in cone mode provides us an opportunity to change the behavior. Leaving these ignored files in the sparse directories makes it impossible to gain performance benefits in the sparse index. When we track into these directories, we need to know if the files are ignored or not, which might depend on the _tracked_ .gitignore file(s) within the sparse directory. This depends on the indexed version of the file, so the sparse directory must be expanded. We must take special care to look for untracked, non-ignored files in these directories before deleting them. We do not want to delete any meaningful work that the users were doing in those directories and perhaps forgot to add and commit before switching sparse-checkout definitions. Since those untracked files might be code files that generated ignored build output, also do not delete any ignored files from these directories in that case. The users can recover their state by resetting their sparse-checkout definition to include that directory and continue. Alternatively, they can see the warning that is presented and delete the directory themselves to regain the performance they expect. By deleting the sparse directories when changing scope (or running 'git sparse-checkout reapply') we regain these performance benefits as if the repository was in a clean state. Since these ignored files are frequently build output or helper files from IDEs, the users should not need the files now that the tracked files are removed. If the tracked files reappear, then they will have newer timestamps than the build artifacts, so the artifacts will need to be regenerated anyway. Use the sparse-index as a data structure in order to find the sparse directories that can be safely deleted. Re-expand the index to a full one if it was full before. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-08 01:42:33 +00:00
clean_tracked_sparse_directories(r);
r->index->sparse_checkout_patterns = NULL;
return result;
}
static char *escaped_pattern(char *pattern)
{
char *p = pattern;
struct strbuf final = STRBUF_INIT;
while (*p) {
if (is_glob_special(*p))
strbuf_addch(&final, '\\');
strbuf_addch(&final, *p);
p++;
}
return strbuf_detach(&final, NULL);
}
static void write_cone_to_file(FILE *fp, struct pattern_list *pl)
{
int i;
struct pattern_entry *pe;
struct hashmap_iter iter;
struct string_list sl = STRING_LIST_INIT_DUP;
struct strbuf parent_pattern = STRBUF_INIT;
hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) {
if (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL))
continue;
if (!hashmap_contains_parent(&pl->recursive_hashmap,
pe->pattern,
&parent_pattern))
string_list_insert(&sl, pe->pattern);
}
string_list_sort(&sl);
string_list_remove_duplicates(&sl, 0);
fprintf(fp, "/*\n!/*/\n");
for (i = 0; i < sl.nr; i++) {
char *pattern = escaped_pattern(sl.items[i].string);
if (strlen(pattern))
fprintf(fp, "%s/\n!%s/*/\n", pattern, pattern);
free(pattern);
}
string_list_clear(&sl, 0);
hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) {
if (!hashmap_contains_parent(&pl->recursive_hashmap,
pe->pattern,
&parent_pattern))
string_list_insert(&sl, pe->pattern);
}
strbuf_release(&parent_pattern);
string_list_sort(&sl);
string_list_remove_duplicates(&sl, 0);
for (i = 0; i < sl.nr; i++) {
char *pattern = escaped_pattern(sl.items[i].string);
fprintf(fp, "%s/\n", pattern);
free(pattern);
}
}
static int write_patterns_and_update(struct pattern_list *pl)
{
char *sparse_filename;
FILE *fp;
int fd;
struct lock_file lk = LOCK_INIT;
int result;
sparse_filename = get_sparse_checkout_filename();
if (safe_create_leading_directories(sparse_filename))
die(_("failed to create directory for sparse-checkout file"));
fd = hold_lock_file_for_update(&lk, sparse_filename,
LOCK_DIE_ON_ERROR);
free(sparse_filename);
result = update_working_directory(pl);
if (result) {
rollback_lock_file(&lk);
clear_pattern_list(pl);
update_working_directory(NULL);
return result;
}
fp = xfdopen(fd, "w");
if (core_sparse_checkout_cone)
write_cone_to_file(fp, pl);
else
write_patterns_to_file(fp, pl);
fflush(fp);
commit_lock_file(&lk);
clear_pattern_list(pl);
return 0;
}
enum sparse_checkout_mode {
MODE_NO_PATTERNS = 0,
MODE_ALL_PATTERNS = 1,
MODE_CONE_PATTERNS = 2,
};
static int set_config(enum sparse_checkout_mode mode)
{
sparse-checkout: set worktree-config correctly `git sparse-checkout set/init` enables worktree-specific configuration[*] by setting extensions.worktreeConfig=true, but neglects to perform the additional necessary bookkeeping of relocating `core.bare=true` and `core.worktree` from $GIT_COMMON_DIR/config to $GIT_COMMON_DIR/config.worktree, as documented in git-worktree.txt. As a result of this oversight, these settings, which are nonsensical for secondary worktrees, can cause Git commands to incorrectly consider a worktree bare (in the case of `core.bare`) or operate on the wrong worktree (in the case of `core.worktree`). Fix this problem by taking advantage of the recently-added init_worktree_config() which enables `extensions.worktreeConfig` and takes care of necessary bookkeeping. While at it, for backward-compatibility reasons, also stop upgrading the repository format to "1" since doing so is (unintentionally) not required to take advantage of `extensions.worktreeConfig`, as explained by 11664196ac ("Revert "check_repository_format_gently(): refuse extensions for old repositories"", 2020-07-15). [*] The main reason to use worktree-specific config for the sparse-checkout builtin was to avoid enabling sparse-checkout patterns in one and causing a loss of files in another. If a worktree does not have a sparse-checkout patterns file, then the sparse-checkout logic will not kick in on that worktree. Reported-by: Sean Allred <allred.sean@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-07 21:33:01 +00:00
/* Update to use worktree config, if not already. */
if (init_worktree_config(the_repository)) {
error(_("failed to initialize worktree config"));
return 1;
}
sparse-checkout: set worktree-config correctly `git sparse-checkout set/init` enables worktree-specific configuration[*] by setting extensions.worktreeConfig=true, but neglects to perform the additional necessary bookkeeping of relocating `core.bare=true` and `core.worktree` from $GIT_COMMON_DIR/config to $GIT_COMMON_DIR/config.worktree, as documented in git-worktree.txt. As a result of this oversight, these settings, which are nonsensical for secondary worktrees, can cause Git commands to incorrectly consider a worktree bare (in the case of `core.bare`) or operate on the wrong worktree (in the case of `core.worktree`). Fix this problem by taking advantage of the recently-added init_worktree_config() which enables `extensions.worktreeConfig` and takes care of necessary bookkeeping. While at it, for backward-compatibility reasons, also stop upgrading the repository format to "1" since doing so is (unintentionally) not required to take advantage of `extensions.worktreeConfig`, as explained by 11664196ac ("Revert "check_repository_format_gently(): refuse extensions for old repositories"", 2020-07-15). [*] The main reason to use worktree-specific config for the sparse-checkout builtin was to avoid enabling sparse-checkout patterns in one and causing a loss of files in another. If a worktree does not have a sparse-checkout patterns file, then the sparse-checkout logic will not kick in on that worktree. Reported-by: Sean Allred <allred.sean@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-07 21:33:01 +00:00
if (repo_config_set_worktree_gently(the_repository,
"core.sparseCheckout",
mode ? "true" : "false") ||
repo_config_set_worktree_gently(the_repository,
"core.sparseCheckoutCone",
mode == MODE_CONE_PATTERNS ?
"true" : "false"))
return 1;
if (mode == MODE_NO_PATTERNS)
sparse-checkout: set worktree-config correctly `git sparse-checkout set/init` enables worktree-specific configuration[*] by setting extensions.worktreeConfig=true, but neglects to perform the additional necessary bookkeeping of relocating `core.bare=true` and `core.worktree` from $GIT_COMMON_DIR/config to $GIT_COMMON_DIR/config.worktree, as documented in git-worktree.txt. As a result of this oversight, these settings, which are nonsensical for secondary worktrees, can cause Git commands to incorrectly consider a worktree bare (in the case of `core.bare`) or operate on the wrong worktree (in the case of `core.worktree`). Fix this problem by taking advantage of the recently-added init_worktree_config() which enables `extensions.worktreeConfig` and takes care of necessary bookkeeping. While at it, for backward-compatibility reasons, also stop upgrading the repository format to "1" since doing so is (unintentionally) not required to take advantage of `extensions.worktreeConfig`, as explained by 11664196ac ("Revert "check_repository_format_gently(): refuse extensions for old repositories"", 2020-07-15). [*] The main reason to use worktree-specific config for the sparse-checkout builtin was to avoid enabling sparse-checkout patterns in one and causing a loss of files in another. If a worktree does not have a sparse-checkout patterns file, then the sparse-checkout logic will not kick in on that worktree. Reported-by: Sean Allred <allred.sean@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-07 21:33:01 +00:00
return set_sparse_index_config(the_repository, 0);
return 0;
}
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
static enum sparse_checkout_mode update_cone_mode(int *cone_mode) {
/* If not specified, use previous definition of cone mode */
if (*cone_mode == -1 && core_apply_sparse_checkout)
*cone_mode = core_sparse_checkout_cone;
/* Set cone/non-cone mode appropriately */
core_apply_sparse_checkout = 1;
if (*cone_mode == 1 || *cone_mode == -1) {
core_sparse_checkout_cone = 1;
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
return MODE_CONE_PATTERNS;
}
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
core_sparse_checkout_cone = 0;
return MODE_ALL_PATTERNS;
}
static int update_modes(int *cone_mode, int *sparse_index)
{
int mode, record_mode;
/* Determine if we need to record the mode; ensure sparse checkout on */
record_mode = (*cone_mode != -1) || !core_apply_sparse_checkout;
mode = update_cone_mode(cone_mode);
if (record_mode && set_config(mode))
return 1;
/* Set sparse-index/non-sparse-index mode if specified */
if (*sparse_index >= 0) {
if (set_sparse_index_config(the_repository, *sparse_index) < 0)
die(_("failed to modify sparse-index config"));
/* force an index rewrite */
repo_read_index(the_repository);
the_repository->index->updated_workdir = 1;
if (!*sparse_index)
ensure_full_index(the_repository->index);
}
return 0;
}
static char const * const builtin_sparse_checkout_init_usage[] = {
"git sparse-checkout init [--cone] [--[no-]sparse-index]",
NULL
};
static struct sparse_checkout_init_opts {
int cone_mode;
int sparse_index;
} init_opts;
static int sparse_checkout_init(int argc, const char **argv, const char *prefix)
{
struct pattern_list pl;
char *sparse_filename;
int res;
struct object_id oid;
struct strbuf pattern = STRBUF_INIT;
static struct option builtin_sparse_checkout_init_options[] = {
OPT_BOOL(0, "cone", &init_opts.cone_mode,
N_("initialize the sparse-checkout in cone mode")),
OPT_BOOL(0, "sparse-index", &init_opts.sparse_index,
N_("toggle the use of a sparse index")),
OPT_END(),
};
setup_work_tree();
repo_read_index(the_repository);
init_opts.cone_mode = -1;
init_opts.sparse_index = -1;
pass subcommand "prefix" arguments to parse_options() Recent commits such as bf0a6b65fc (builtin/multi-pack-index.c: let parse-options parse subcommands, 2022-08-19) converted a few functions to match our usual argc/argv/prefix conventions, but the prefix argument remains unused. However, there is a good use for it: they should pass it to their own parse_options() functions, where it may be used to adjust the value of any filename options. In all but one of these functions, there's no behavior change, since they don't use OPT_FILENAME. But this is an actual fix for one option, which you can see by modifying the test suite like so: diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 4fe57414c1..d0974d4371 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -186,7 +186,11 @@ test_expect_success 'writing a bitmap with --refs-snapshot' ' # Then again, but with a refs snapshot which only sees # refs/tags/one. - git multi-pack-index write --bitmap --refs-snapshot=snapshot && + ( + mkdir subdir && + cd subdir && + git multi-pack-index write --bitmap --refs-snapshot=../snapshot + ) && test_path_is_file $midx && test_path_is_file $midx-$(midx_checksum $objdir).bitmap && I'd emphasize that this wasn't broken by bf0a6b65fc; it has been broken all along, because the sub-function never got to see the prefix. It is that commit which is actually enabling us to fix it (and which also brought attention to the problem because it triggers -Wunused-parameter!) The other functions changed here don't use OPT_FILENAME at all. In their cases this isn't fixing anything visible, but it's following the usual pattern and future-proofing them against somebody adding new options and being surprised. I didn't include a test for the one visible case above. We don't generally test routine parse-options behavior for individual options. The challenge here was finding the problem, and now that this has been done, it's not likely to regress. Likewise, we could apply the patch above to cover it "for free" but it makes reading the rest of the test unnecessarily complicated. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-25 10:47:00 +00:00
argc = parse_options(argc, argv, prefix,
builtin_sparse_checkout_init_options,
builtin_sparse_checkout_init_usage, 0);
if (update_modes(&init_opts.cone_mode, &init_opts.sparse_index))
return 1;
memset(&pl, 0, sizeof(pl));
sparse_filename = get_sparse_checkout_filename();
res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL, 0);
/* If we already have a sparse-checkout file, use it. */
if (res >= 0) {
free(sparse_filename);
return update_working_directory(NULL);
}
if (repo_get_oid(the_repository, "HEAD", &oid)) {
FILE *fp;
/* assume we are in a fresh repo, but update the sparse-checkout file */
if (safe_create_leading_directories(sparse_filename))
die(_("unable to create leading directories of %s"),
sparse_filename);
fp = xfopen(sparse_filename, "w");
if (!fp)
die(_("failed to open '%s'"), sparse_filename);
free(sparse_filename);
fprintf(fp, "/*\n!/*/\n");
fclose(fp);
return 0;
}
strbuf_addstr(&pattern, "/*");
add_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0);
strbuf_addstr(&pattern, "!/*/");
add_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0);
pl.use_cone_patterns = init_opts.cone_mode;
return write_patterns_and_update(&pl);
}
static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)
{
struct pattern_entry *e = xmalloc(sizeof(*e));
e->patternlen = path->len;
e->pattern = strbuf_detach(path, NULL);
hashmap_entry_init(&e->ent, fspathhash(e->pattern));
hashmap_add(&pl->recursive_hashmap, &e->ent);
while (e->patternlen) {
char *slash = strrchr(e->pattern, '/');
char *oldpattern = e->pattern;
size_t newlen;
if (!slash || slash == e->pattern)
break;
newlen = slash - e->pattern;
e = xmalloc(sizeof(struct pattern_entry));
e->patternlen = newlen;
e->pattern = xstrndup(oldpattern, newlen);
hashmap_entry_init(&e->ent, fspathhash(e->pattern));
if (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL))
hashmap_add(&pl->parent_hashmap, &e->ent);
}
}
static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)
{
strbuf_trim(line);
strbuf_trim_trailing_dir_sep(line);
if (strbuf_normalize_path(line))
die(_("could not normalize path %s"), line->buf);
if (!line->len)
return;
if (line->buf[0] != '/')
strbuf_insertstr(line, 0, "/");
insert_recursive_pattern(pl, line);
}
static void add_patterns_from_input(struct pattern_list *pl,
int argc, const char **argv,
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
FILE *file)
{
int i;
if (core_sparse_checkout_cone) {
struct strbuf line = STRBUF_INIT;
hashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);
hashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);
pl->use_cone_patterns = 1;
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
if (file) {
struct strbuf unquoted = STRBUF_INIT;
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
while (!strbuf_getline(&line, file)) {
if (line.buf[0] == '"') {
strbuf_reset(&unquoted);
if (unquote_c_style(&unquoted, line.buf, NULL))
die(_("unable to unquote C-style string '%s'"),
line.buf);
strbuf_swap(&unquoted, &line);
}
strbuf_to_cone_pattern(&line, pl);
}
strbuf_release(&unquoted);
} else {
for (i = 0; i < argc; i++) {
strbuf_setlen(&line, 0);
strbuf_addstr(&line, argv[i]);
strbuf_to_cone_pattern(&line, pl);
}
}
} else {
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
if (file) {
struct strbuf line = STRBUF_INIT;
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
while (!strbuf_getline(&line, file)) {
size_t len;
char *buf = strbuf_detach(&line, &len);
add_pattern(buf, empty_base, 0, pl, 0);
}
} else {
for (i = 0; i < argc; i++)
add_pattern(argv[i], empty_base, 0, pl, 0);
}
}
}
enum modify_type {
REPLACE,
ADD,
};
static void add_patterns_cone_mode(int argc, const char **argv,
struct pattern_list *pl,
int use_stdin)
{
struct strbuf buffer = STRBUF_INIT;
struct pattern_entry *pe;
struct hashmap_iter iter;
struct pattern_list existing;
char *sparse_filename = get_sparse_checkout_filename();
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
add_patterns_from_input(pl, argc, argv,
use_stdin ? stdin : NULL);
memset(&existing, 0, sizeof(existing));
existing.use_cone_patterns = core_sparse_checkout_cone;
if (add_patterns_from_file_to_list(sparse_filename, "", 0,
&existing, NULL, 0))
die(_("unable to load existing sparse-checkout patterns"));
free(sparse_filename);
if (!existing.use_cone_patterns)
die(_("existing sparse-checkout patterns do not use cone mode"));
hashmap_for_each_entry(&existing.recursive_hashmap, &iter, pe, ent) {
if (!hashmap_contains_parent(&pl->recursive_hashmap,
pe->pattern, &buffer) ||
!hashmap_contains_parent(&pl->parent_hashmap,
pe->pattern, &buffer)) {
strbuf_reset(&buffer);
strbuf_addstr(&buffer, pe->pattern);
insert_recursive_pattern(pl, &buffer);
}
}
clear_pattern_list(&existing);
strbuf_release(&buffer);
}
static void add_patterns_literal(int argc, const char **argv,
struct pattern_list *pl,
int use_stdin)
{
char *sparse_filename = get_sparse_checkout_filename();
if (add_patterns_from_file_to_list(sparse_filename, "", 0,
pl, NULL, 0))
die(_("unable to load existing sparse-checkout patterns"));
free(sparse_filename);
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
add_patterns_from_input(pl, argc, argv, use_stdin ? stdin : NULL);
}
static int modify_pattern_list(int argc, const char **argv, int use_stdin,
enum modify_type m)
{
int result;
int changed_config = 0;
struct pattern_list *pl = xcalloc(1, sizeof(*pl));
switch (m) {
case ADD:
if (core_sparse_checkout_cone)
add_patterns_cone_mode(argc, argv, pl, use_stdin);
else
add_patterns_literal(argc, argv, pl, use_stdin);
break;
case REPLACE:
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
add_patterns_from_input(pl, argc, argv,
use_stdin ? stdin : NULL);
break;
}
if (!core_apply_sparse_checkout) {
set_config(MODE_ALL_PATTERNS);
core_apply_sparse_checkout = 1;
changed_config = 1;
}
result = write_patterns_and_update(pl);
if (result && changed_config)
set_config(MODE_NO_PATTERNS);
clear_pattern_list(pl);
free(pl);
return result;
}
static void sanitize_paths(int argc, const char **argv,
const char *prefix, int skip_checks)
sparse-checkout: pay attention to prefix for {set, add} In cone mode, non-option arguments to set & add are clearly paths, and as such, we should pay attention to prefix. In non-cone mode, it is not clear that folks intend to provide paths since the inputs are gitignore-style patterns. Paying attention to prefix would prevent folks from doing things like git sparse-checkout add /.gitattributes git sparse-checkout add '/toplevel-dir/*' In fact, the former will result in fatal: '/.gitattributes' is outside repository... while the later will result in fatal: Invalid path '/toplevel-dir': No such file or directory despite the fact that both are valid gitignore-style patterns that would select real files if added to the sparse-checkout file. This might lead people to just use the path without the leading slash, potentially resulting in them grabbing files with the same name throughout the directory hierarchy contrary to their expectations. See also [1] and [2]. Adding prefix seems to just be fraught with error; so for now simply throw an error in non-cone mode when sparse-checkout set/add are run from a subdirectory. [1] https://lore.kernel.org/git/e1934710-e228-adc4-d37c-f706883bd27c@gmail.com/ [2] https://lore.kernel.org/git/CABPp-BHXZ-XLxY0a3wCATfdq=6-EjW62RzbxKAoFPeXfJswD2w@mail.gmail.com/ Helped-by: Junio Hamano <gitster@pobox.com> Reviewed-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-19 16:44:43 +00:00
{
int i;
sparse-checkout: pay attention to prefix for {set, add} In cone mode, non-option arguments to set & add are clearly paths, and as such, we should pay attention to prefix. In non-cone mode, it is not clear that folks intend to provide paths since the inputs are gitignore-style patterns. Paying attention to prefix would prevent folks from doing things like git sparse-checkout add /.gitattributes git sparse-checkout add '/toplevel-dir/*' In fact, the former will result in fatal: '/.gitattributes' is outside repository... while the later will result in fatal: Invalid path '/toplevel-dir': No such file or directory despite the fact that both are valid gitignore-style patterns that would select real files if added to the sparse-checkout file. This might lead people to just use the path without the leading slash, potentially resulting in them grabbing files with the same name throughout the directory hierarchy contrary to their expectations. See also [1] and [2]. Adding prefix seems to just be fraught with error; so for now simply throw an error in non-cone mode when sparse-checkout set/add are run from a subdirectory. [1] https://lore.kernel.org/git/e1934710-e228-adc4-d37c-f706883bd27c@gmail.com/ [2] https://lore.kernel.org/git/CABPp-BHXZ-XLxY0a3wCATfdq=6-EjW62RzbxKAoFPeXfJswD2w@mail.gmail.com/ Helped-by: Junio Hamano <gitster@pobox.com> Reviewed-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-19 16:44:43 +00:00
if (!argc)
return;
if (prefix && *prefix && core_sparse_checkout_cone) {
/*
* The args are not pathspecs, so unfortunately we
* cannot imitate how cmd_add() uses parse_pathspec().
*/
int prefix_len = strlen(prefix);
for (i = 0; i < argc; i++)
argv[i] = prefix_path(prefix, prefix_len, argv[i]);
}
if (skip_checks)
return;
sparse-checkout: pay attention to prefix for {set, add} In cone mode, non-option arguments to set & add are clearly paths, and as such, we should pay attention to prefix. In non-cone mode, it is not clear that folks intend to provide paths since the inputs are gitignore-style patterns. Paying attention to prefix would prevent folks from doing things like git sparse-checkout add /.gitattributes git sparse-checkout add '/toplevel-dir/*' In fact, the former will result in fatal: '/.gitattributes' is outside repository... while the later will result in fatal: Invalid path '/toplevel-dir': No such file or directory despite the fact that both are valid gitignore-style patterns that would select real files if added to the sparse-checkout file. This might lead people to just use the path without the leading slash, potentially resulting in them grabbing files with the same name throughout the directory hierarchy contrary to their expectations. See also [1] and [2]. Adding prefix seems to just be fraught with error; so for now simply throw an error in non-cone mode when sparse-checkout set/add are run from a subdirectory. [1] https://lore.kernel.org/git/e1934710-e228-adc4-d37c-f706883bd27c@gmail.com/ [2] https://lore.kernel.org/git/CABPp-BHXZ-XLxY0a3wCATfdq=6-EjW62RzbxKAoFPeXfJswD2w@mail.gmail.com/ Helped-by: Junio Hamano <gitster@pobox.com> Reviewed-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-19 16:44:43 +00:00
if (prefix && *prefix && !core_sparse_checkout_cone)
die(_("please run from the toplevel directory in non-cone mode"));
if (core_sparse_checkout_cone) {
for (i = 0; i < argc; i++) {
if (argv[i][0] == '/')
die(_("specify directories rather than patterns (no leading slash)"));
if (argv[i][0] == '!')
die(_("specify directories rather than patterns. If your directory starts with a '!', pass --skip-checks"));
if (strpbrk(argv[i], "*?[]"))
die(_("specify directories rather than patterns. If your directory really has any of '*?[]\\' in it, pass --skip-checks"));
}
}
for (i = 0; i < argc; i++) {
struct cache_entry *ce;
struct index_state *index = the_repository->index;
int pos = index_name_pos(index, argv[i], strlen(argv[i]));
if (pos < 0)
continue;
ce = index->cache[pos];
if (S_ISSPARSEDIR(ce->ce_mode))
continue;
if (core_sparse_checkout_cone)
die(_("'%s' is not a directory; to treat it as a directory anyway, rerun with --skip-checks"), argv[i]);
else
warning(_("pass a leading slash before paths such as '%s' if you want a single file (see NON-CONE PROBLEMS in the git-sparse-checkout manual)."), argv[i]);
}
sparse-checkout: pay attention to prefix for {set, add} In cone mode, non-option arguments to set & add are clearly paths, and as such, we should pay attention to prefix. In non-cone mode, it is not clear that folks intend to provide paths since the inputs are gitignore-style patterns. Paying attention to prefix would prevent folks from doing things like git sparse-checkout add /.gitattributes git sparse-checkout add '/toplevel-dir/*' In fact, the former will result in fatal: '/.gitattributes' is outside repository... while the later will result in fatal: Invalid path '/toplevel-dir': No such file or directory despite the fact that both are valid gitignore-style patterns that would select real files if added to the sparse-checkout file. This might lead people to just use the path without the leading slash, potentially resulting in them grabbing files with the same name throughout the directory hierarchy contrary to their expectations. See also [1] and [2]. Adding prefix seems to just be fraught with error; so for now simply throw an error in non-cone mode when sparse-checkout set/add are run from a subdirectory. [1] https://lore.kernel.org/git/e1934710-e228-adc4-d37c-f706883bd27c@gmail.com/ [2] https://lore.kernel.org/git/CABPp-BHXZ-XLxY0a3wCATfdq=6-EjW62RzbxKAoFPeXfJswD2w@mail.gmail.com/ Helped-by: Junio Hamano <gitster@pobox.com> Reviewed-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-19 16:44:43 +00:00
}
static char const * const builtin_sparse_checkout_add_usage[] = {
N_("git sparse-checkout add [--skip-checks] (--stdin | <patterns>)"),
NULL
};
static struct sparse_checkout_add_opts {
int skip_checks;
int use_stdin;
} add_opts;
static int sparse_checkout_add(int argc, const char **argv, const char *prefix)
{
static struct option builtin_sparse_checkout_add_options[] = {
OPT_BOOL_F(0, "skip-checks", &add_opts.skip_checks,
N_("skip some sanity checks on the given paths that might give false positives"),
PARSE_OPT_NONEG),
OPT_BOOL(0, "stdin", &add_opts.use_stdin,
N_("read patterns from standard in")),
OPT_END(),
};
setup_work_tree();
if (!core_apply_sparse_checkout)
die(_("no sparse-checkout to add to"));
repo_read_index(the_repository);
argc = parse_options(argc, argv, prefix,
builtin_sparse_checkout_add_options,
builtin_sparse_checkout_add_usage,
PARSE_OPT_KEEP_UNKNOWN_OPT);
sanitize_paths(argc, argv, prefix, add_opts.skip_checks);
sparse-checkout: pay attention to prefix for {set, add} In cone mode, non-option arguments to set & add are clearly paths, and as such, we should pay attention to prefix. In non-cone mode, it is not clear that folks intend to provide paths since the inputs are gitignore-style patterns. Paying attention to prefix would prevent folks from doing things like git sparse-checkout add /.gitattributes git sparse-checkout add '/toplevel-dir/*' In fact, the former will result in fatal: '/.gitattributes' is outside repository... while the later will result in fatal: Invalid path '/toplevel-dir': No such file or directory despite the fact that both are valid gitignore-style patterns that would select real files if added to the sparse-checkout file. This might lead people to just use the path without the leading slash, potentially resulting in them grabbing files with the same name throughout the directory hierarchy contrary to their expectations. See also [1] and [2]. Adding prefix seems to just be fraught with error; so for now simply throw an error in non-cone mode when sparse-checkout set/add are run from a subdirectory. [1] https://lore.kernel.org/git/e1934710-e228-adc4-d37c-f706883bd27c@gmail.com/ [2] https://lore.kernel.org/git/CABPp-BHXZ-XLxY0a3wCATfdq=6-EjW62RzbxKAoFPeXfJswD2w@mail.gmail.com/ Helped-by: Junio Hamano <gitster@pobox.com> Reviewed-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-19 16:44:43 +00:00
return modify_pattern_list(argc, argv, add_opts.use_stdin, ADD);
}
static char const * const builtin_sparse_checkout_set_usage[] = {
N_("git sparse-checkout set [--[no-]cone] [--[no-]sparse-index] [--skip-checks] (--stdin | <patterns>)"),
NULL
};
static struct sparse_checkout_set_opts {
int cone_mode;
int sparse_index;
int skip_checks;
int use_stdin;
} set_opts;
static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
{
int default_patterns_nr = 2;
const char *default_patterns[] = {"/*", "!/*/", NULL};
static struct option builtin_sparse_checkout_set_options[] = {
OPT_BOOL(0, "cone", &set_opts.cone_mode,
N_("initialize the sparse-checkout in cone mode")),
OPT_BOOL(0, "sparse-index", &set_opts.sparse_index,
N_("toggle the use of a sparse index")),
OPT_BOOL_F(0, "skip-checks", &set_opts.skip_checks,
N_("skip some sanity checks on the given paths that might give false positives"),
PARSE_OPT_NONEG),
OPT_BOOL_F(0, "stdin", &set_opts.use_stdin,
N_("read patterns from standard in"),
PARSE_OPT_NONEG),
OPT_END(),
};
setup_work_tree();
repo_read_index(the_repository);
set_opts.cone_mode = -1;
set_opts.sparse_index = -1;
argc = parse_options(argc, argv, prefix,
builtin_sparse_checkout_set_options,
builtin_sparse_checkout_set_usage,
PARSE_OPT_KEEP_UNKNOWN_OPT);
if (update_modes(&set_opts.cone_mode, &set_opts.sparse_index))
return 1;
/*
* Cone mode automatically specifies the toplevel directory. For
* non-cone mode, if nothing is specified, manually select just the
* top-level directory (much as 'init' would do).
*/
if (!core_sparse_checkout_cone && argc == 0) {
argv = default_patterns;
argc = default_patterns_nr;
sparse-checkout: pay attention to prefix for {set, add} In cone mode, non-option arguments to set & add are clearly paths, and as such, we should pay attention to prefix. In non-cone mode, it is not clear that folks intend to provide paths since the inputs are gitignore-style patterns. Paying attention to prefix would prevent folks from doing things like git sparse-checkout add /.gitattributes git sparse-checkout add '/toplevel-dir/*' In fact, the former will result in fatal: '/.gitattributes' is outside repository... while the later will result in fatal: Invalid path '/toplevel-dir': No such file or directory despite the fact that both are valid gitignore-style patterns that would select real files if added to the sparse-checkout file. This might lead people to just use the path without the leading slash, potentially resulting in them grabbing files with the same name throughout the directory hierarchy contrary to their expectations. See also [1] and [2]. Adding prefix seems to just be fraught with error; so for now simply throw an error in non-cone mode when sparse-checkout set/add are run from a subdirectory. [1] https://lore.kernel.org/git/e1934710-e228-adc4-d37c-f706883bd27c@gmail.com/ [2] https://lore.kernel.org/git/CABPp-BHXZ-XLxY0a3wCATfdq=6-EjW62RzbxKAoFPeXfJswD2w@mail.gmail.com/ Helped-by: Junio Hamano <gitster@pobox.com> Reviewed-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-19 16:44:43 +00:00
} else {
sanitize_paths(argc, argv, prefix, set_opts.skip_checks);
}
return modify_pattern_list(argc, argv, set_opts.use_stdin, REPLACE);
}
static char const * const builtin_sparse_checkout_reapply_usage[] = {
"git sparse-checkout reapply [--[no-]cone] [--[no-]sparse-index]",
NULL
};
static struct sparse_checkout_reapply_opts {
int cone_mode;
int sparse_index;
} reapply_opts;
static int sparse_checkout_reapply(int argc, const char **argv,
const char *prefix)
{
static struct option builtin_sparse_checkout_reapply_options[] = {
OPT_BOOL(0, "cone", &reapply_opts.cone_mode,
N_("initialize the sparse-checkout in cone mode")),
OPT_BOOL(0, "sparse-index", &reapply_opts.sparse_index,
N_("toggle the use of a sparse index")),
OPT_END(),
};
setup_work_tree();
if (!core_apply_sparse_checkout)
die(_("must be in a sparse-checkout to reapply sparsity patterns"));
reapply_opts.cone_mode = -1;
reapply_opts.sparse_index = -1;
pass subcommand "prefix" arguments to parse_options() Recent commits such as bf0a6b65fc (builtin/multi-pack-index.c: let parse-options parse subcommands, 2022-08-19) converted a few functions to match our usual argc/argv/prefix conventions, but the prefix argument remains unused. However, there is a good use for it: they should pass it to their own parse_options() functions, where it may be used to adjust the value of any filename options. In all but one of these functions, there's no behavior change, since they don't use OPT_FILENAME. But this is an actual fix for one option, which you can see by modifying the test suite like so: diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 4fe57414c1..d0974d4371 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -186,7 +186,11 @@ test_expect_success 'writing a bitmap with --refs-snapshot' ' # Then again, but with a refs snapshot which only sees # refs/tags/one. - git multi-pack-index write --bitmap --refs-snapshot=snapshot && + ( + mkdir subdir && + cd subdir && + git multi-pack-index write --bitmap --refs-snapshot=../snapshot + ) && test_path_is_file $midx && test_path_is_file $midx-$(midx_checksum $objdir).bitmap && I'd emphasize that this wasn't broken by bf0a6b65fc; it has been broken all along, because the sub-function never got to see the prefix. It is that commit which is actually enabling us to fix it (and which also brought attention to the problem because it triggers -Wunused-parameter!) The other functions changed here don't use OPT_FILENAME at all. In their cases this isn't fixing anything visible, but it's following the usual pattern and future-proofing them against somebody adding new options and being surprised. I didn't include a test for the one visible case above. We don't generally test routine parse-options behavior for individual options. The challenge here was finding the problem, and now that this has been done, it's not likely to regress. Likewise, we could apply the patch above to cover it "for free" but it makes reading the rest of the test unnecessarily complicated. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-25 10:47:00 +00:00
argc = parse_options(argc, argv, prefix,
builtin_sparse_checkout_reapply_options,
builtin_sparse_checkout_reapply_usage, 0);
repo_read_index(the_repository);
if (update_modes(&reapply_opts.cone_mode, &reapply_opts.sparse_index))
return 1;
return update_working_directory(NULL);
}
static char const * const builtin_sparse_checkout_disable_usage[] = {
"git sparse-checkout disable",
NULL
};
static int sparse_checkout_disable(int argc, const char **argv,
const char *prefix)
{
static struct option builtin_sparse_checkout_disable_options[] = {
OPT_END(),
};
struct pattern_list pl;
struct strbuf match_all = STRBUF_INIT;
/*
* We do not exit early if !core_apply_sparse_checkout; due to the
* ability for users to manually muck things up between
* direct editing of .git/info/sparse-checkout
* running read-tree -m u HEAD or update-index --skip-worktree
* direct toggling of config options
* users might end up with an index with SKIP_WORKTREE bit set on
* some files and not know how to undo it. So, here we just
* forcibly return to a dense checkout regardless of initial state.
*/
setup_work_tree();
pass subcommand "prefix" arguments to parse_options() Recent commits such as bf0a6b65fc (builtin/multi-pack-index.c: let parse-options parse subcommands, 2022-08-19) converted a few functions to match our usual argc/argv/prefix conventions, but the prefix argument remains unused. However, there is a good use for it: they should pass it to their own parse_options() functions, where it may be used to adjust the value of any filename options. In all but one of these functions, there's no behavior change, since they don't use OPT_FILENAME. But this is an actual fix for one option, which you can see by modifying the test suite like so: diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 4fe57414c1..d0974d4371 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -186,7 +186,11 @@ test_expect_success 'writing a bitmap with --refs-snapshot' ' # Then again, but with a refs snapshot which only sees # refs/tags/one. - git multi-pack-index write --bitmap --refs-snapshot=snapshot && + ( + mkdir subdir && + cd subdir && + git multi-pack-index write --bitmap --refs-snapshot=../snapshot + ) && test_path_is_file $midx && test_path_is_file $midx-$(midx_checksum $objdir).bitmap && I'd emphasize that this wasn't broken by bf0a6b65fc; it has been broken all along, because the sub-function never got to see the prefix. It is that commit which is actually enabling us to fix it (and which also brought attention to the problem because it triggers -Wunused-parameter!) The other functions changed here don't use OPT_FILENAME at all. In their cases this isn't fixing anything visible, but it's following the usual pattern and future-proofing them against somebody adding new options and being surprised. I didn't include a test for the one visible case above. We don't generally test routine parse-options behavior for individual options. The challenge here was finding the problem, and now that this has been done, it's not likely to regress. Likewise, we could apply the patch above to cover it "for free" but it makes reading the rest of the test unnecessarily complicated. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-25 10:47:00 +00:00
argc = parse_options(argc, argv, prefix,
builtin_sparse_checkout_disable_options,
builtin_sparse_checkout_disable_usage, 0);
repo_read_index(the_repository);
memset(&pl, 0, sizeof(pl));
hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);
hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);
pl.use_cone_patterns = 0;
core_apply_sparse_checkout = 1;
strbuf_addstr(&match_all, "/*");
add_pattern(strbuf_detach(&match_all, NULL), empty_base, 0, &pl, 0);
prepare_repo_settings(the_repository);
the_repository->settings.sparse_index = 0;
if (update_working_directory(&pl))
die(_("error while refreshing working directory"));
clear_pattern_list(&pl);
return set_config(MODE_NO_PATTERNS);
}
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
static char const * const builtin_sparse_checkout_check_rules_usage[] = {
N_("git sparse-checkout check-rules [-z] [--skip-checks]"
"[--[no-]cone] [--rules-file <file>]"),
NULL
};
static struct sparse_checkout_check_rules_opts {
int cone_mode;
int null_termination;
char *rules_file;
} check_rules_opts;
static int check_rules(struct pattern_list *pl, int null_terminated) {
struct strbuf line = STRBUF_INIT;
struct strbuf unquoted = STRBUF_INIT;
char *path;
int line_terminator = null_terminated ? 0 : '\n';
strbuf_getline_fn getline_fn = null_terminated ? strbuf_getline_nul
: strbuf_getline;
the_repository->index->sparse_checkout_patterns = pl;
while (!getline_fn(&line, stdin)) {
path = line.buf;
if (!null_terminated && line.buf[0] == '"') {
strbuf_reset(&unquoted);
if (unquote_c_style(&unquoted, line.buf, NULL))
die(_("unable to unquote C-style string '%s'"),
line.buf);
path = unquoted.buf;
}
if (path_in_sparse_checkout(path, the_repository->index))
write_name_quoted(path, stdout, line_terminator);
}
strbuf_release(&line);
strbuf_release(&unquoted);
return 0;
}
static int sparse_checkout_check_rules(int argc, const char **argv, const char *prefix)
{
static struct option builtin_sparse_checkout_check_rules_options[] = {
OPT_BOOL('z', NULL, &check_rules_opts.null_termination,
N_("terminate input and output files by a NUL character")),
OPT_BOOL(0, "cone", &check_rules_opts.cone_mode,
N_("when used with --rules-file interpret patterns as cone mode patterns")),
OPT_FILENAME(0, "rules-file", &check_rules_opts.rules_file,
N_("use patterns in <file> instead of the current ones.")),
OPT_END(),
};
FILE *fp;
int ret;
struct pattern_list pl = {0};
char *sparse_filename;
check_rules_opts.cone_mode = -1;
argc = parse_options(argc, argv, prefix,
builtin_sparse_checkout_check_rules_options,
builtin_sparse_checkout_check_rules_usage,
PARSE_OPT_KEEP_UNKNOWN_OPT);
if (check_rules_opts.rules_file && check_rules_opts.cone_mode < 0)
check_rules_opts.cone_mode = 1;
update_cone_mode(&check_rules_opts.cone_mode);
pl.use_cone_patterns = core_sparse_checkout_cone;
if (check_rules_opts.rules_file) {
fp = xfopen(check_rules_opts.rules_file, "r");
add_patterns_from_input(&pl, argc, argv, fp);
fclose(fp);
} else {
sparse_filename = get_sparse_checkout_filename();
if (add_patterns_from_file_to_list(sparse_filename, "", 0, &pl,
NULL, 0))
die(_("unable to load existing sparse-checkout patterns"));
free(sparse_filename);
}
ret = check_rules(&pl, check_rules_opts.null_termination);
clear_pattern_list(&pl);
return ret;
}
int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)
{
parse_opt_subcommand_fn *fn = NULL;
struct option builtin_sparse_checkout_options[] = {
OPT_SUBCOMMAND("list", &fn, sparse_checkout_list),
OPT_SUBCOMMAND("init", &fn, sparse_checkout_init),
OPT_SUBCOMMAND("set", &fn, sparse_checkout_set),
OPT_SUBCOMMAND("add", &fn, sparse_checkout_add),
OPT_SUBCOMMAND("reapply", &fn, sparse_checkout_reapply),
OPT_SUBCOMMAND("disable", &fn, sparse_checkout_disable),
builtin/sparse-checkout: add check-rules command There exists no direct way to interrogate git about which paths are matched by a given set of sparsity rules. It is possible to get this information from git, but it includes checking out the commit that contains the paths, applying the sparse checkout patterns and then using something like 'git ls-files -t' to check if the skip worktree bit is set. This works in some case, but there are cases where it is awkward or infeasible to generate a checkout for this purpose. Exposing the pattern matching of sparse checkout enables more tooling to be built and avoids a situation where tools that want to reason about sparse checkouts start containing parallel implementation of the rules. To accommodate this, add a 'check-rules' subcommand to the 'sparse-checkout' builtin along the lines of the 'git check-ignore' and 'git check-attr' commands. The new command accepts a list of paths on stdin and outputs just the ones the match the sparse checkout. To allow for use in a bare repository and to allow for interrogating about other patterns than the current ones, include a '--rules-file' option which allows the caller to explicitly pass sparse checkout rules in the format accepted by 'sparse-checkout set --stdin'. To allow for reuse of the handling of input patterns for the '--rules-file' flag, modify 'add_patterns_from_input()' to be able to read from a 'FILE' instead of just stdin. To allow for reuse of the logic which decides whether or not rules should be interpreted as cone-mode patterns, split that part out of 'update_modes()' such that can be called without modifying the config. An alternative could have been to create a new 'check-sparsity' command. However, placing it under 'sparse-checkout' allows for a) more easily re-using the sparse checkout pattern matching and cone/non-code mode handling, and b) keeps the documentation for the command next to the experimental warning and the cone-mode discussion. Signed-off-by: William Sprent <williams@unity3d.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-27 07:55:03 +00:00
OPT_SUBCOMMAND("check-rules", &fn, sparse_checkout_check_rules),
OPT_END(),
};
argc = parse_options(argc, argv, prefix,
builtin_sparse_checkout_options,
builtin_sparse_checkout_usage, 0);
git_config(git_default_config, NULL);
sparse-checkout: integrate with sparse index When modifying the sparse-checkout definition, the sparse-checkout builtin calls update_sparsity() to modify the SKIP_WORKTREE bits of all cache entries in the index. Before, we needed the index to be fully expanded in order to ensure we had the full list of files necessary that match the new patterns. Insert a call to reset_sparse_directories() that expands sparse directories that are within the new pattern list, but only far enough that every necessary file path now exists as a cache entry. The remaining logic within update_sparsity() will modify the SKIP_WORKTREE bits appropriately. This allows us to disable command_requires_full_index within the sparse-checkout builtin. Add tests that demonstrate that we are not expanding to a full index unnecessarily. We can see the improved performance in the p2000 test script: Test HEAD~1 HEAD ------------------------------------------------------------------------ 2000.24: git ... (sparse-v3) 2.14(1.55+0.58) 1.57(1.03+0.53) -26.6% 2000.25: git ... (sparse-v4) 2.20(1.62+0.57) 1.58(0.98+0.59) -28.2% These reductions of 26-28% are small compared to most examples, but the time is dominated by writing a new copy of the base repository to the worktree and then deleting it again. The fact that the previous index expansion was such a large portion of the time is telling how important it is to complete this sparse index integration. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-23 13:48:46 +00:00
prepare_repo_settings(the_repository);
the_repository->settings.command_requires_full_index = 0;
return fn(argc, argv, prefix);
}