git/t/t7063-status-untracked-cache.sh

995 lines
27 KiB
Bash
Raw Normal View History

#!/bin/sh
test_description='test untracked cache'
GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
tests: mark tests relying on the current default for `init.defaultBranch` In addition to the manual adjustment to let the `linux-gcc` CI job run the test suite with `master` and then with `main`, this patch makes sure that GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME is set in all test scripts that currently rely on the initial branch name being `master by default. To determine which test scripts to mark up, the first step was to force-set the default branch name to `master` in - all test scripts that contain the keyword `master`, - t4211, which expects `t/t4211/history.export` with a hard-coded ref to initialize the default branch, - t5560 because it sources `t/t556x_common` which uses `master`, - t8002 and t8012 because both source `t/annotate-tests.sh` which also uses `master`) This trick was performed by this command: $ sed -i '/^ *\. \.\/\(test-lib\|lib-\(bash\|cvs\|git-svn\)\|gitweb-lib\)\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' $(git grep -l master t/t[0-9]*.sh) \ t/t4211*.sh t/t5560*.sh t/t8002*.sh t/t8012*.sh After that, careful, manual inspection revealed that some of the test scripts containing the needle `master` do not actually rely on a specific default branch name: either they mention `master` only in a comment, or they initialize that branch specificially, or they do not actually refer to the current default branch. Therefore, the aforementioned modification was undone in those test scripts thusly: $ git checkout HEAD -- \ t/t0027-auto-crlf.sh t/t0060-path-utils.sh \ t/t1011-read-tree-sparse-checkout.sh \ t/t1305-config-include.sh t/t1309-early-config.sh \ t/t1402-check-ref-format.sh t/t1450-fsck.sh \ t/t2024-checkout-dwim.sh \ t/t2106-update-index-assume-unchanged.sh \ t/t3040-subprojects-basic.sh t/t3301-notes.sh \ t/t3308-notes-merge.sh t/t3423-rebase-reword.sh \ t/t3436-rebase-more-options.sh \ t/t4015-diff-whitespace.sh t/t4257-am-interactive.sh \ t/t5323-pack-redundant.sh t/t5401-update-hooks.sh \ t/t5511-refspec.sh t/t5526-fetch-submodules.sh \ t/t5529-push-errors.sh t/t5530-upload-pack-error.sh \ t/t5548-push-porcelain.sh \ t/t5552-skipping-fetch-negotiator.sh \ t/t5572-pull-submodule.sh t/t5608-clone-2gb.sh \ t/t5614-clone-submodules-shallow.sh \ t/t7508-status.sh t/t7606-merge-custom.sh \ t/t9302-fast-import-unpack-limit.sh We excluded one set of test scripts in these commands, though: the range of `git p4` tests. The reason? `git p4` stores the (foreign) remote branch in the branch called `p4/master`, which is obviously not the default branch. Manual analysis revealed that only five of these tests actually require a specific default branch name to pass; They were modified thusly: $ sed -i '/^ *\. \.\/lib-git-p4\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' t/t980[0167]*.sh t/t9811*.sh Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-18 23:44:19 +00:00
export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
. ./test-lib.sh
t7063: work around FreeBSD's lazy mtime update feature Let's start with the commit message of [1] from freebsd.git [2] Sync timestamp changes for inodes of special files to disk as late as possible (when the inode is reclaimed). Temporarily only do this if option UFS_LAZYMOD configured and softupdates aren't enabled. UFS_LAZYMOD is intentionally left out of /sys/conf/options. This is mainly to avoid almost useless disk i/o on battery powered machines. It's silly to write to disk (on the next sync or when the inode becomes inactive) just because someone hit a key or something wrote to the screen or /dev/null. PR: 5577 [3] The short version of that, in the context of t7063, is that when a directory is updated, its mtime may be updated later, not immediately. This can be shown with a simple command sequence date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd . One would expect that the date shown in `ls` would be one second from `date`, but it's 10 seconds later. If we put another `ls -lTd .` in front of `sleep 10`, then the date of the last `ls` comes as expected. The first `ls` somehow forces mtime to be updated. t7063 is really sensitive to directory mtime. When mtime is too "new", git code suspects racy timestamps and will not trigger the shortcut in untracked cache, in t7063.24 and eventually be detected in t7063.27 We have two options thanks to this special FreeBSD feature: 1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely when running on FreeBSD 2) Work around this problem (using the same 'ls' trick) and continue to support untracked cache on FreeBSD I initially wanted to go with 1) because I didn't know the exact nature of this feature and feared that it would make untracked cache work unreliably, using the cached version when it should not. Since the behavior of this thing is clearer now. The picture is not that bad. If this indeed happens often, untracked cache would assume racy condition more often and _fall back_ to non-untracked cache code paths. Which means it may be less effective, but it will not show wrong things. This patch goes with option 2. PS. For those who want to look further in FreeBSD source code, this flag is now called IN_LAZYMOD. I can see it's effective in ext2 and ufs. zfs is not affected. [1] 660e6408e6df99a20dacb070c5e7f9739efdf96d [2] git://github.com/freebsd/freebsd.git [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577 Reported-by: Eric Wong <e@80x24.org> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-03 17:45:22 +00:00
# On some filesystems (e.g. FreeBSD's ext2 and ufs) directory mtime
# is updated lazily after contents in the directory changes, which
# forces the untracked cache code to take the slow path. A test
# that wants to make sure that the fast path works correctly should
# call this helper to make mtime of the containing directory in sync
# with the reality before checking the fast path behaviour.
#
# See <20160803174522.5571-1-pclouds@gmail.com> if you want to know
# more.
GIT_FORCE_UNTRACKED_CACHE=true
export GIT_FORCE_UNTRACKED_CACHE
t7063: work around FreeBSD's lazy mtime update feature Let's start with the commit message of [1] from freebsd.git [2] Sync timestamp changes for inodes of special files to disk as late as possible (when the inode is reclaimed). Temporarily only do this if option UFS_LAZYMOD configured and softupdates aren't enabled. UFS_LAZYMOD is intentionally left out of /sys/conf/options. This is mainly to avoid almost useless disk i/o on battery powered machines. It's silly to write to disk (on the next sync or when the inode becomes inactive) just because someone hit a key or something wrote to the screen or /dev/null. PR: 5577 [3] The short version of that, in the context of t7063, is that when a directory is updated, its mtime may be updated later, not immediately. This can be shown with a simple command sequence date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd . One would expect that the date shown in `ls` would be one second from `date`, but it's 10 seconds later. If we put another `ls -lTd .` in front of `sleep 10`, then the date of the last `ls` comes as expected. The first `ls` somehow forces mtime to be updated. t7063 is really sensitive to directory mtime. When mtime is too "new", git code suspects racy timestamps and will not trigger the shortcut in untracked cache, in t7063.24 and eventually be detected in t7063.27 We have two options thanks to this special FreeBSD feature: 1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely when running on FreeBSD 2) Work around this problem (using the same 'ls' trick) and continue to support untracked cache on FreeBSD I initially wanted to go with 1) because I didn't know the exact nature of this feature and feared that it would make untracked cache work unreliably, using the cached version when it should not. Since the behavior of this thing is clearer now. The picture is not that bad. If this indeed happens often, untracked cache would assume racy condition more often and _fall back_ to non-untracked cache code paths. Which means it may be less effective, but it will not show wrong things. This patch goes with option 2. PS. For those who want to look further in FreeBSD source code, this flag is now called IN_LAZYMOD. I can see it's effective in ext2 and ufs. zfs is not affected. [1] 660e6408e6df99a20dacb070c5e7f9739efdf96d [2] git://github.com/freebsd/freebsd.git [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577 Reported-by: Eric Wong <e@80x24.org> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-03 17:45:22 +00:00
sync_mtime () {
find . -type d -exec ls -ld {} + >/dev/null
t7063: work around FreeBSD's lazy mtime update feature Let's start with the commit message of [1] from freebsd.git [2] Sync timestamp changes for inodes of special files to disk as late as possible (when the inode is reclaimed). Temporarily only do this if option UFS_LAZYMOD configured and softupdates aren't enabled. UFS_LAZYMOD is intentionally left out of /sys/conf/options. This is mainly to avoid almost useless disk i/o on battery powered machines. It's silly to write to disk (on the next sync or when the inode becomes inactive) just because someone hit a key or something wrote to the screen or /dev/null. PR: 5577 [3] The short version of that, in the context of t7063, is that when a directory is updated, its mtime may be updated later, not immediately. This can be shown with a simple command sequence date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd . One would expect that the date shown in `ls` would be one second from `date`, but it's 10 seconds later. If we put another `ls -lTd .` in front of `sleep 10`, then the date of the last `ls` comes as expected. The first `ls` somehow forces mtime to be updated. t7063 is really sensitive to directory mtime. When mtime is too "new", git code suspects racy timestamps and will not trigger the shortcut in untracked cache, in t7063.24 and eventually be detected in t7063.27 We have two options thanks to this special FreeBSD feature: 1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely when running on FreeBSD 2) Work around this problem (using the same 'ls' trick) and continue to support untracked cache on FreeBSD I initially wanted to go with 1) because I didn't know the exact nature of this feature and feared that it would make untracked cache work unreliably, using the cached version when it should not. Since the behavior of this thing is clearer now. The picture is not that bad. If this indeed happens often, untracked cache would assume racy condition more often and _fall back_ to non-untracked cache code paths. Which means it may be less effective, but it will not show wrong things. This patch goes with option 2. PS. For those who want to look further in FreeBSD source code, this flag is now called IN_LAZYMOD. I can see it's effective in ext2 and ufs. zfs is not affected. [1] 660e6408e6df99a20dacb070c5e7f9739efdf96d [2] git://github.com/freebsd/freebsd.git [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577 Reported-by: Eric Wong <e@80x24.org> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-03 17:45:22 +00:00
}
avoid_racy() {
sleep 1
}
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
status_is_clean() {
git status --porcelain >../status.actual &&
test_must_be_empty ../status.actual
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
}
# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can
# compare commands side-by-side, e.g.
# iuc status --porcelain >expect &&
# git status --porcelain >actual &&
# test_cmp expect actual
iuc () {
git ls-files -s >../current-index-entries
git ls-files -t | sed -ne s/^S.//p >../current-sparse-entries
GIT_INDEX_FILE=.git/tmp_index
export GIT_INDEX_FILE
git update-index --index-info <../current-index-entries
git update-index --skip-worktree $(cat ../current-sparse-entries)
git -c core.untrackedCache=false "$@"
ret=$?
rm ../current-index-entries
rm $GIT_INDEX_FILE
unset GIT_INDEX_FILE
return $ret
}
get_relevant_traces () {
# From the GIT_TRACE2_PERF data of the form
# $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT
# extract the $RELEVANT_STAT fields. We don't care about region_enter
# or region_leave, or stats for things outside read_directory.
INPUT_FILE=$1
OUTPUT_FILE=$2
grep data.*read_directo $INPUT_FILE |
cut -d "|" -f 9 |
grep -v visited \
>"$OUTPUT_FILE"
}
test_lazy_prereq UNTRACKED_CACHE '
config: add core.untrackedCache When we know that mtime on directory as given by the environment is usable for the purpose of untracked cache, we may want the untracked cache to be always used without any mtime test or kernel name check being performed. Also when we know that mtime is not usable for the purpose of untracked cache, for example because the repo is shared over a network file system, we may want the untracked-cache to be automatically removed from the index. Allow the user to express such preference by setting the 'core.untrackedCache' configuration variable, which can take 'keep', 'false', or 'true' and default to 'keep'. When read_index_from() is called, it now adds or removes the untracked cache in the index to respect the value of this variable. So it does nothing if the value is `keep` or if the variable is unset; it adds the untracked cache if the value is `true`; and it removes the cache if the value is `false`. `git update-index --[no-|force-]untracked-cache` still adds the untracked cache to, or removes it, from the index, but this shows a warning if it goes against the value of core.untrackedCache, because the next time the index is read the untracked cache will be added or removed if the configuration is set to do so. Also `--untracked-cache` used to check that the underlying operating system and file system change `st_mtime` field of a directory if files are added or deleted in that directory. But because those tests take a long time, `--untracked-cache` no longer performs them. Instead, there is now `--test-untracked-cache` to perform the tests. This change makes `--untracked-cache` the same as `--force-untracked-cache`. This last change is backward incompatible and should be mentioned in the release notes. Helped-by: Duy Nguyen <pclouds@gmail.com> Helped-by: Torsten Bögershausen <tboegi@web.de> Helped-by: Stefan Beller <sbeller@google.com> Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> read-cache: Duy'sfixup Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-01-27 06:58:05 +00:00
{ git update-index --test-untracked-cache; ret=$?; } &&
test $ret -ne 1
'
if ! test_have_prereq UNTRACKED_CACHE; then
skip_all='This system does not support untracked cache'
test_done
fi
test_expect_success 'core.untrackedCache is unset' '
test_must_fail git config --get core.untrackedCache
'
test_expect_success 'setup' '
git init --template= worktree &&
cd worktree &&
mkdir done dtwo dthree &&
touch one two three done/one dtwo/two dthree/three &&
test-tool chmtime =-300 one two three done/one dtwo/two dthree/three &&
test-tool chmtime =-300 done dtwo dthree &&
test-tool chmtime =-300 . &&
git add one two done/one &&
mkdir .git/info &&
: >.git/info/exclude &&
git update-index --untracked-cache &&
test_oid_cache <<-EOF
root sha1:e6fcc8f2ee31bae321d66afd183fcb7237afae6e
root sha256:b90c672088c015b9c83876e919da311bad4cd39639fb139f988af6a11493b974
exclude sha1:13263c0978fb9fad16b2d580fb800b6d811c3ff0
exclude sha256:fe4aaa1bbbbce4cb8f73426748a14c5ad6026b26f90505a0bf2494b165a5b76c
done sha1:1946f0437f90c5005533cbe1736a6451ca301714
done sha256:7f079501d79f665b3acc50f5e0e9e94509084d5032ac20113a37dd5029b757cc
EOF
'
test_expect_success 'untracked cache is empty' '
test-tool dump-untracked-cache >../actual &&
cat >../expect-empty <<EOF &&
info/exclude $ZERO_OID
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
EOF
test_cmp ../expect-empty ../actual
'
cat >../status.expect <<EOF &&
A done/one
A one
A two
?? dthree/
?? dtwo/
?? three
EOF
cat >../dump.expect <<EOF &&
info/exclude $EMPTY_BLOB
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $ZERO_OID recurse valid
dthree/
dtwo/
three
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
three
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_expect_success 'status first time (empty cache)' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:3
....gitignore-invalidation:1
....directory-invalidation:0
....opendir:4
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'untracked cache after first status' '
test-tool dump-untracked-cache >../actual &&
test_cmp ../dump.expect ../actual
'
test_expect_success 'status second time (fully populated cache)' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:0
....directory-invalidation:0
....opendir:0
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'untracked cache after second status' '
test-tool dump-untracked-cache >../actual &&
test_cmp ../dump.expect ../actual
'
cat >../status_uall.expect <<EOF &&
A done/one
A one
A two
?? dthree/three
?? dtwo/two
?? three
EOF
# Bypassing the untracked cache here is not desirable from an
# end-user perspective, but is expected in the current design.
# The untracked cache data stored for a -unormal run cannot be
# correctly used in a -uall run - it would yield incorrect output.
test_expect_success 'untracked cache is bypassed with -uall' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status -uall --porcelain >../actual &&
iuc status -uall --porcelain >../status.iuc &&
test_cmp ../status_uall.expect ../status.iuc &&
test_cmp ../status_uall.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'untracked cache remains after bypass' '
test-tool dump-untracked-cache >../actual &&
test_cmp ../dump.expect ../actual
'
untracked-cache: support '--untracked-files=all' if configured Untracked cache was originally designed to only work with "--untracked-files=normal", and is bypassed when "--untracked-files=all" is requested, but this causes performance issues for UI tooling that wants to see "all" on a frequent basis. On the other hand, the conditions that altogether prevented applicability to the "all" mode no longer seem to apply, after several major refactors in recent years; this possibility was discussed in 81153d02-8e7a-be59-e709-e90cd5906f3a@jeffhostetler.com and CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com, and somewhat confirmed experimentally by several users using a version of this patch to use untracked cache with -uall for about a year. When 'git status' runs without using the untracked cache, on a large repo, on windows, with fsmonitor, it can run very slowly. This can make GUIs that need to use "-uall" (and therefore currently bypass untracked cache) unusable when fsmonitor is enabled, on such large repos. To partially address this, align the supported directory flags for the stored untracked cache data with the git config. If a user specifies an '--untracked-files=' commandline parameter that does not align with their 'status.showuntrackedfiles' config value, then the untracked cache will be ignored - as it is for other unsupported situations like when a pathspec is specified. If the previously stored flags no longer match the current configuration, but the currently-applicable flags do match the current configuration, then discard the previously stored untracked cache data. For most users there will be no change in behavior. Users who need '--untracked-files=all' to perform well will now have the option of setting "status.showuntrackedfiles" to "all" for better / more consistent performance. Users who need '--untracked-files=all' to perform well for their tooling AND prefer to avoid the verbosity of "all" when running git status explicitly without options... are out of luck for now (no change). Users who have the "status.showuntrackedfiles" config set to "all" and yet frequently explicitly call 'git status --untracked-files=normal' (and use the untracked cache) are the only ones who will be disadvantaged by this change. Their "--untracked-files=normal" calls will, after this change, no longer use the untracked cache. Signed-off-by: Tao Klerks <tao@klerks.biz> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-31 16:02:15 +00:00
test_expect_success 'if -uall is configured, untracked cache gets populated by default' '
test_config status.showuntrackedfiles all &&
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
test_cmp ../status_uall.expect ../status.iuc &&
test_cmp ../status_uall.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:3
....gitignore-invalidation:1
....directory-invalidation:0
....opendir:4
EOF
test_cmp ../trace.expect ../trace.relevant
'
cat >../dump_uall.expect <<EOF &&
info/exclude $EMPTY_BLOB
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000000
/ $ZERO_OID recurse valid
three
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse valid
three
/dtwo/ $ZERO_OID recurse valid
two
EOF
test_expect_success 'if -uall was configured, untracked cache is populated' '
test-tool dump-untracked-cache >../actual &&
test_cmp ../dump_uall.expect ../actual
'
test_expect_success 'if -uall is configured, untracked cache is used by default' '
test_config status.showuntrackedfiles all &&
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
test_cmp ../status_uall.expect ../status.iuc &&
test_cmp ../status_uall.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:0
....directory-invalidation:0
....opendir:0
EOF
test_cmp ../trace.expect ../trace.relevant
'
# Bypassing the untracked cache here is not desirable from an
# end-user perspective, but is expected in the current design.
# The untracked cache data stored for a -all run cannot be
# correctly used in a -unormal run - it would yield incorrect
# output.
test_expect_success 'if -uall is configured, untracked cache is bypassed with -unormal' '
test_config status.showuntrackedfiles all &&
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status -unormal --porcelain >../actual &&
iuc status -unormal --porcelain >../status.iuc &&
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'repopulate untracked cache for -unormal' '
git status --porcelain
'
test_expect_success 'modify in root directory, one dir invalidation' '
: >four &&
test-tool chmtime =-240 four &&
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
A done/one
A one
A two
?? dthree/
?? dtwo/
?? four
?? three
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:0
....directory-invalidation:1
....opendir:1
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'verify untracked cache dump' '
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $EMPTY_BLOB
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $ZERO_OID recurse valid
dthree/
dtwo/
four
three
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
three
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'new .gitignore invalidates recursively' '
echo four >.gitignore &&
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
A done/one
A one
A two
?? .gitignore
?? dthree/
?? dtwo/
?? three
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:1
....directory-invalidation:1
....opendir:4
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'verify untracked cache dump' '
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $EMPTY_BLOB
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse valid
.gitignore
dthree/
dtwo/
three
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
three
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'new info/exclude invalidates everything' '
echo three >>.git/info/exclude &&
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
A done/one
A one
A two
?? .gitignore
?? dtwo/
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:1
....directory-invalidation:0
....opendir:4
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'verify untracked cache dump' '
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $(test_oid exclude)
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse valid
.gitignore
dtwo/
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'move two from tracked to untracked' '
git rm --cached two &&
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $(test_oid exclude)
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'status after the move' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
A done/one
A one
?? .gitignore
?? dtwo/
?? two
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:0
....directory-invalidation:0
....opendir:1
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'verify untracked cache dump' '
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $(test_oid exclude)
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse valid
.gitignore
dtwo/
two
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'move two from untracked to tracked' '
git add two &&
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $(test_oid exclude)
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'status after the move' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
A done/one
A one
A two
?? .gitignore
?? dtwo/
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:0
....directory-invalidation:0
....opendir:1
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'verify untracked cache dump' '
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $(test_oid exclude)
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse valid
.gitignore
dtwo/
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'set up for sparse checkout testing' '
echo two >done/.gitignore &&
echo three >>done/.gitignore &&
echo two >done/two &&
git add -f done/two done/.gitignore &&
git commit -m "first commit"
'
test_expect_success 'status after commit' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
?? .gitignore
?? dtwo/
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:0
....directory-invalidation:0
....opendir:2
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'untracked cache correct after commit' '
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $(test_oid exclude)
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse valid
.gitignore
dtwo/
/done/ $ZERO_OID recurse valid
/dthree/ $ZERO_OID recurse check_only valid
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'set up sparse checkout' '
echo "done/[a-z]*" >.git/info/sparse-checkout &&
test_config core.sparsecheckout true &&
git checkout main &&
git update-index --force-untracked-cache &&
git status --porcelain >/dev/null && # prime the cache
test_path_is_missing done/.gitignore &&
test_path_is_file done/one
'
test_expect_success 'create/modify files, some of which are gitignored' '
echo two bis >done/two &&
echo three >done/three && # three is gitignored
echo four >done/four && # four is gitignored at a higher level
echo five >done/five && # five is not gitignored
test-tool chmtime =-180 done/two done/three done/four done/five done &&
# we need to ensure that the root dir is touched (in the past);
test-tool chmtime =-180 . &&
t7063: work around FreeBSD's lazy mtime update feature Let's start with the commit message of [1] from freebsd.git [2] Sync timestamp changes for inodes of special files to disk as late as possible (when the inode is reclaimed). Temporarily only do this if option UFS_LAZYMOD configured and softupdates aren't enabled. UFS_LAZYMOD is intentionally left out of /sys/conf/options. This is mainly to avoid almost useless disk i/o on battery powered machines. It's silly to write to disk (on the next sync or when the inode becomes inactive) just because someone hit a key or something wrote to the screen or /dev/null. PR: 5577 [3] The short version of that, in the context of t7063, is that when a directory is updated, its mtime may be updated later, not immediately. This can be shown with a simple command sequence date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd . One would expect that the date shown in `ls` would be one second from `date`, but it's 10 seconds later. If we put another `ls -lTd .` in front of `sleep 10`, then the date of the last `ls` comes as expected. The first `ls` somehow forces mtime to be updated. t7063 is really sensitive to directory mtime. When mtime is too "new", git code suspects racy timestamps and will not trigger the shortcut in untracked cache, in t7063.24 and eventually be detected in t7063.27 We have two options thanks to this special FreeBSD feature: 1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely when running on FreeBSD 2) Work around this problem (using the same 'ls' trick) and continue to support untracked cache on FreeBSD I initially wanted to go with 1) because I didn't know the exact nature of this feature and feared that it would make untracked cache work unreliably, using the cached version when it should not. Since the behavior of this thing is clearer now. The picture is not that bad. If this indeed happens often, untracked cache would assume racy condition more often and _fall back_ to non-untracked cache code paths. Which means it may be less effective, but it will not show wrong things. This patch goes with option 2. PS. For those who want to look further in FreeBSD source code, this flag is now called IN_LAZYMOD. I can see it's effective in ext2 and ufs. zfs is not affected. [1] 660e6408e6df99a20dacb070c5e7f9739efdf96d [2] git://github.com/freebsd/freebsd.git [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577 Reported-by: Eric Wong <e@80x24.org> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-03 17:45:22 +00:00
sync_mtime
'
test_expect_success 'test sparse status with untracked cache' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../status.actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
M done/two
?? .gitignore
?? done/five
?? dtwo/
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../status.actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:1
....directory-invalidation:2
....opendir:2
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'untracked cache correct after status' '
test-tool dump-untracked-cache >../actual &&
cat >../expect <<EOF &&
info/exclude $(test_oid exclude)
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse valid
.gitignore
dtwo/
/done/ $(test_oid done) recurse valid
five
/dthree/ $ZERO_OID recurse check_only valid
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect ../actual
'
test_expect_success 'test sparse status again with untracked cache' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../status.actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
M done/two
?? .gitignore
?? done/five
?? dtwo/
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../status.actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:0
....directory-invalidation:0
....opendir:0
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'set up for test of subdir and sparse checkouts' '
mkdir done/sub &&
mkdir done/sub/sub &&
echo "sub" > done/sub/sub/file &&
test-tool chmtime =-120 done/sub/sub/file done/sub/sub done/sub done
'
test_expect_success 'test sparse status with untracked cache and subdir' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../status.actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
M done/two
?? .gitignore
?? done/five
?? done/sub/
?? dtwo/
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../status.actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:2
....gitignore-invalidation:0
....directory-invalidation:1
....opendir:3
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'verify untracked cache dump (sparse/subdirs)' '
test-tool dump-untracked-cache >../actual &&
cat >../expect-from-test-dump <<EOF &&
info/exclude $(test_oid exclude)
core.excludesfile $ZERO_OID
exclude_per_dir .gitignore
flags 00000006
/ $(test_oid root) recurse valid
.gitignore
dtwo/
/done/ $(test_oid done) recurse valid
five
sub/
/done/sub/ $ZERO_OID recurse check_only valid
sub/
/done/sub/sub/ $ZERO_OID recurse check_only valid
file
/dthree/ $ZERO_OID recurse check_only valid
/dtwo/ $ZERO_OID recurse check_only valid
two
EOF
test_cmp ../expect-from-test-dump ../actual
'
test_expect_success 'test sparse status again with untracked cache and subdir' '
: >../trace.output &&
GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \
git status --porcelain >../status.actual &&
iuc status --porcelain >../status.iuc &&
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../status.actual &&
get_relevant_traces ../trace.output ../trace.relevant &&
cat >../trace.expect <<EOF &&
....path:
....node-creation:0
....gitignore-invalidation:0
....directory-invalidation:0
....opendir:0
EOF
test_cmp ../trace.expect ../trace.relevant
'
test_expect_success 'move entry in subdir from untracked to cached' '
git add dtwo/two &&
git status --porcelain >../status.actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
M done/two
A dtwo/two
?? .gitignore
?? done/five
?? done/sub/
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../status.actual
'
test_expect_success 'move entry in subdir from cached to untracked' '
git rm --cached dtwo/two &&
git status --porcelain >../status.actual &&
iuc status --porcelain >../status.iuc &&
cat >../status.expect <<EOF &&
M done/two
?? .gitignore
?? done/five
?? done/sub/
?? dtwo/
EOF
test_cmp ../status.expect ../status.iuc &&
test_cmp ../status.expect ../status.actual
'
test_expect_success '--no-untracked-cache removes the cache' '
git update-index --no-untracked-cache &&
test-tool dump-untracked-cache >../actual &&
echo "no untracked cache" >../expect-no-uc &&
test_cmp ../expect-no-uc ../actual
'
test_expect_success 'git status does not change anything' '
git status &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-no-uc ../actual
'
test_expect_success 'setting core.untrackedCache to true and using git status creates the cache' '
git config core.untrackedCache true &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-no-uc ../actual &&
git status &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-from-test-dump ../actual
'
test_expect_success 'using --no-untracked-cache does not fail when core.untrackedCache is true' '
git update-index --no-untracked-cache &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-no-uc ../actual &&
git update-index --untracked-cache &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-empty ../actual
'
test_expect_success 'setting core.untrackedCache to false and using git status removes the cache' '
git config core.untrackedCache false &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-empty ../actual &&
git status &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-no-uc ../actual
'
test_expect_success 'using --untracked-cache does not fail when core.untrackedCache is false' '
git update-index --untracked-cache &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-empty ../actual
'
test_expect_success 'setting core.untrackedCache to keep' '
git config core.untrackedCache keep &&
git update-index --untracked-cache &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-empty ../actual &&
git status &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-from-test-dump ../actual &&
git update-index --no-untracked-cache &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-no-uc ../actual &&
git update-index --force-untracked-cache &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-empty ../actual &&
git status &&
test-tool dump-untracked-cache >../actual &&
test_cmp ../expect-from-test-dump ../actual
'
test_expect_success 'test ident field is working' '
mkdir ../other_worktree &&
cp -R done dthree dtwo four three ../other_worktree &&
GIT_WORK_TREE=../other_worktree git status 2>../err &&
echo "warning: untracked cache is disabled on this system or location" >../expect &&
test_cmp ../expect ../err
'
test_expect_success 'untracked cache survives a checkout' '
git commit --allow-empty -m empty &&
test-tool dump-untracked-cache >../before &&
test_when_finished "git checkout main" &&
git checkout -b other_branch &&
test-tool dump-untracked-cache >../after &&
test_cmp ../before ../after &&
test_commit test &&
test-tool dump-untracked-cache >../before &&
git checkout main &&
test-tool dump-untracked-cache >../after &&
test_cmp ../before ../after
'
test_expect_success 'untracked cache survives a commit' '
test-tool dump-untracked-cache >../before &&
git add done/two &&
git commit -m commit &&
test-tool dump-untracked-cache >../after &&
test_cmp ../before ../after
'
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
test_expect_success 'teardown worktree' '
cd ..
'
test_expect_success SYMLINKS 'setup worktree for symlink test' '
git init worktree-symlink &&
cd worktree-symlink &&
git config core.untrackedCache true &&
mkdir one two &&
touch one/file two/file &&
git add one/file two/file &&
git commit -m"first commit" &&
git rm -rf one &&
ln -s two one &&
git add one &&
git commit -m"second commit"
'
dir.c: fix missing dir invalidation in untracked code Let's start with how create a new directory cache after the last one becomes invalid (e.g. because its dir mtime has changed...). In open_cached_dir(): 1. We start out with valid_cached_dir() returning false, which should call invalidate_directory() to put a directory state back to initial state, no untracked entries (untracked_nr zero), no sub directory traversal (dirs[].recurse zero). 2. Since the cache cannot be used, we go the slow path opendir() and go through items one by one via readdir(). All the directories on disk will be added back to the cache (if not already exist in dirs[]) and its flag "recurse" gets changed to one to note that it's part of the cached dir travesal next time. 3. By the time we reach close_cached_dir() we should have a good subdir list in dirs[]. Those with "recurse" flag set are the ones present in the on-disk directory. The directory is now marked "valid". Next time read_directory() is called, since the directory is marked valid, it will skip readdir(), go fast path and traverse through dirs[] array instead. Steps one and two need some tight cooperation. If a subdir is removed, readdir() will not find it and of course we cannot examine/invalidate it. To make sure removed directories on disk are gone from the cache, step one must make sure recurse flag of all subdirs are zero. But that's not true. If "valid" flag is already false, there is a chance we go straight to the end of valid_cached_dir() without calling invalidate_directory(). Or we fail to meet the "if (untracked-valid)" condition and skip over the invalidate_directory(). After step 3, we mark the cache valid. Any stale subdir with incorrect recurse flag becomes a real subdir next time we traverse the directory using dirs[] array. We could avoid this by making sure invalidate_directory() is always called (therefore dirs[].recurse cleared) at the beginning of open_cached_dir(). Which is what this patch does. As to how we get into this situation, the key in the test is this command git checkout master where "one/file" is replaced with "one" in the index. This index update triggers untracked_cache_invalidate_path(), which clears valid flag of the root directory while keeping "recurse" flag on the subdir "one" on. On the next git-status, we go through steps 1-3 above and save an incorrect cache on disk. The second git-status blindly follows the bad cache data and shows the problem. This is arguably because of a bad design where "recurse" flag plays double roles: whether a directory should be saved on disk, and whether it is part of a directory traversal. We need to keep recurse flag set at "checkout master" because of the first role: we need to keep subdir caches (dir "two" for example has not been touched at all, no reason to throw its cache away). As long as we make sure to ignore/reset "recurse" flag at the beginning of a directory traversal, we're good. But maybe eventually we should separate these two roles. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:21 +00:00
test_expect_success SYMLINKS '"status" after symlink replacement should be clean with UC=true' '
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
git checkout HEAD~ &&
status_is_clean &&
status_is_clean &&
git checkout main &&
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
avoid_racy &&
status_is_clean &&
status_is_clean
'
test_expect_success SYMLINKS '"status" after symlink replacement should be clean with UC=false' '
git config core.untrackedCache false &&
git checkout HEAD~ &&
status_is_clean &&
status_is_clean &&
git checkout main &&
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
avoid_racy &&
status_is_clean &&
status_is_clean
'
test_expect_success 'setup worktree for non-symlink test' '
git init worktree-non-symlink &&
cd worktree-non-symlink &&
git config core.untrackedCache true &&
mkdir one two &&
touch one/file two/file &&
git add one/file two/file &&
git commit -m"first commit" &&
git rm -rf one &&
cp two/file one &&
git add one &&
git commit -m"second commit"
'
dir.c: fix missing dir invalidation in untracked code Let's start with how create a new directory cache after the last one becomes invalid (e.g. because its dir mtime has changed...). In open_cached_dir(): 1. We start out with valid_cached_dir() returning false, which should call invalidate_directory() to put a directory state back to initial state, no untracked entries (untracked_nr zero), no sub directory traversal (dirs[].recurse zero). 2. Since the cache cannot be used, we go the slow path opendir() and go through items one by one via readdir(). All the directories on disk will be added back to the cache (if not already exist in dirs[]) and its flag "recurse" gets changed to one to note that it's part of the cached dir travesal next time. 3. By the time we reach close_cached_dir() we should have a good subdir list in dirs[]. Those with "recurse" flag set are the ones present in the on-disk directory. The directory is now marked "valid". Next time read_directory() is called, since the directory is marked valid, it will skip readdir(), go fast path and traverse through dirs[] array instead. Steps one and two need some tight cooperation. If a subdir is removed, readdir() will not find it and of course we cannot examine/invalidate it. To make sure removed directories on disk are gone from the cache, step one must make sure recurse flag of all subdirs are zero. But that's not true. If "valid" flag is already false, there is a chance we go straight to the end of valid_cached_dir() without calling invalidate_directory(). Or we fail to meet the "if (untracked-valid)" condition and skip over the invalidate_directory(). After step 3, we mark the cache valid. Any stale subdir with incorrect recurse flag becomes a real subdir next time we traverse the directory using dirs[] array. We could avoid this by making sure invalidate_directory() is always called (therefore dirs[].recurse cleared) at the beginning of open_cached_dir(). Which is what this patch does. As to how we get into this situation, the key in the test is this command git checkout master where "one/file" is replaced with "one" in the index. This index update triggers untracked_cache_invalidate_path(), which clears valid flag of the root directory while keeping "recurse" flag on the subdir "one" on. On the next git-status, we go through steps 1-3 above and save an incorrect cache on disk. The second git-status blindly follows the bad cache data and shows the problem. This is arguably because of a bad design where "recurse" flag plays double roles: whether a directory should be saved on disk, and whether it is part of a directory traversal. We need to keep recurse flag set at "checkout master" because of the first role: we need to keep subdir caches (dir "two" for example has not been touched at all, no reason to throw its cache away). As long as we make sure to ignore/reset "recurse" flag at the beginning of a directory traversal, we're good. But maybe eventually we should separate these two roles. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:21 +00:00
test_expect_success '"status" after file replacement should be clean with UC=true' '
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
git checkout HEAD~ &&
status_is_clean &&
status_is_clean &&
git checkout main &&
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
avoid_racy &&
status_is_clean &&
test-tool dump-untracked-cache >../actual &&
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
grep -F "recurse valid" ../actual >../actual.grep &&
cat >../expect.grep <<EOF &&
/ $ZERO_OID recurse valid
/two/ $ZERO_OID recurse valid
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
EOF
status_is_clean &&
test_cmp ../expect.grep ../actual.grep
'
test_expect_success '"status" after file replacement should be clean with UC=false' '
git config core.untrackedCache false &&
git checkout HEAD~ &&
status_is_clean &&
status_is_clean &&
git checkout main &&
status: add a failing test showing a core.untrackedCache bug The untracked cache gets confused when a directory is swapped out for a file. It is easiest to reproduce this by swapping out a directory with a symlink to another directory, and as the tests show the symlink case is the only case we've found where "git status" will subsequently report incorrect information, even though it's possible to otherwise get the untracked cache into a state where its internal data structures don't reflect reality. In the symlink case, whatever files are inside the target of the symlink will be incorrectly shown as untracked. This issue does not happen if the symlink links to another file, only if it links to another directory. A stand-alone testcase for copying into a terminal: ( rm -rf /tmp/testrepo && git init /tmp/testrepo && cd /tmp/testrepo && mkdir x y && touch x/a y/b && git add x/a y/b && git commit -msnap && git rm -rf y && ln -s x y && git add y && git commit -msnap2 && git checkout HEAD~ && git status && git checkout master && sleep 1 && git status && git status ) This will incorrectly show y/a as an untracked file. Both the "git status" call right before "git checkout master" and the "sleep 1" after the "checkout master" are needed to reproduce this, presumably due to the untracked cache tracking on the basis of cached whole seconds from stat(2). When git gets into this state, a workaround to fix it is to issue a one-off: git -c core.untrackedCache=false status For the non-symlink case, the bug is that the output of test-dump-untracked-cache should not include: /one/ 0000000000000000000000000000000000000000 recurse valid It being in the output implies that cached traversal of root includes the directory "one" which does not exist on disk anymore. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-24 09:30:19 +00:00
avoid_racy &&
status_is_clean &&
status_is_clean
'
test_expect_success 'empty repo (no index) and core.untrackedCache' '
git init emptyrepo &&
git -C emptyrepo -c core.untrackedCache=true write-tree
'
test_done