Commit graph

67869 commits

Author SHA1 Message Date
Junio C Hamano bc820cf9e6 Merge branch 'vd/scalar-enables-fsmonitor'
"scalar" now enables built-in fsmonitor on enlisted repositories,
when able.

* vd/scalar-enables-fsmonitor:
  scalar: update technical doc roadmap with FSMonitor support
  scalar unregister: stop FSMonitor daemon
  scalar: enable built-in FSMonitor on `register`
  scalar: move config setting logic into its own function
  scalar-delete: do not 'die()' in 'delete_enlistment()'
  scalar-[un]register: clearly indicate source of error
  scalar-unregister: handle error codes greater than 0
  scalar: constrain enlistment search
2022-08-29 14:55:12 -07:00
Junio C Hamano 0b08ba7eb6 Merge branch 'en/ancestry-path-in-a-range'
"git rev-list --ancestry-path=C A..B" is a natural extension of
"git rev-list A..B"; instead of choosing a subset of A..B to those
that have ancestry relationship with A, it lets a subset with
ancestry relationship with C.

* en/ancestry-path-in-a-range:
  revision: allow --ancestry-path to take an argument
  t6019: modernize tests with helper
  rev-list-options.txt: fix simple typo
2022-08-29 14:55:11 -07:00
Junio C Hamano 64cb4c34d1 Merge branch 'mt/rot13-in-c'
Test portability improvements.

* mt/rot13-in-c:
  tests: use the new C rot13-filter helper to avoid PERL prereq
  t0021: implementation the rot13-filter.pl script in C
  t0021: avoid grepping for a Perl-specific string at filter output
2022-08-29 14:55:11 -07:00
Junio C Hamano c068a3b8ee Merge branch 'ds/decorate-filter-tweak'
The namespaces used by "log --decorate" from "refs/" hierarchy by
default has been tightened.

* ds/decorate-filter-tweak:
  fetch: use ref_namespaces during prefetch
  maintenance: stop writing log.excludeDecoration
  log: create log.initialDecorationSet=all
  log: add --clear-decorations option
  log: add default decoration filter
  log-tree: use ref_namespaces instead of if/else-if
  refs: use ref_namespaces for replace refs base
  refs: add array of ref namespaces
  t4207: test coloring of grafted decorations
  t4207: modernize test
  refs: allow "HEAD" as decoration filter
2022-08-29 14:55:11 -07:00
Eric Sunshine 3871bdb7e4 t4301: emit blank line in more idiomatic fashion
The unusual use of:

    printf "\\n" >>file &&

may give readers pause, making them wonder why this form was chosen over
the more typical:

    printf "\n" >>file &&

However, even that may give pause since it is a somewhat unusual and
long-winded way of saying:

    echo >>file &&

Therefore, replace `printf` with the more idiomatic `echo`, with the
hope of eliminating a possible stumbling block for those reading the
code.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-29 09:28:40 -07:00
Eric Sunshine 87ed97167a t4301: fix broken &&-chains and add missing loop termination
Fix &&-chain breaks in a couple tests which went unnoticed due to blind
spots in the &&-chain linters. In particular, the "magic exit code 117"
&&-chain checker built into test-lib.sh only recognizes broken &&-chains
at the top-level; it does not work within `{...}` groups, `(...)`
subshells, `$(...)` substitutions, or within bodies of compound
statements, such as `if`, `for`, `while`, `case`, etc. Furthermore,
`chainlint.sed`, which detects broken &&-chains only in `(...)`
subshells, missed these cases (which are in subshells) because it
(surprisingly) neglects to check for intact &&-chain on single-line
`for` loops.

While at it, explicitly signal failure of commands within the `for`
loops (which might arise due to the filesystem being full or "inode"
exhaustion). This is important since failures within `for` and `while`
loops can go unnoticed if not detected and signaled manually since the
loop itself does not abort when a contained command fails, nor will a
failure necessarily be detected when the loop finishes since the loop
returns the exit code of the last command it ran on the final iteration,
which may not be the command which failed.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-29 09:28:31 -07:00
brian m. carlson ee69e7884e gc: use temporary file for editing crontab
While cron is specified by POSIX, there are a wide variety of
implementations in use.  "git maintenance" assumes that the
"crontab" command can be fed from its standard input the new
contents and the syntax to do so is not to have any filename
argument, as POSIX describes.  However, on FreeBSD, the cron
implementation requires a file name argument: if the user wants to
edit standard input, they must specify "-".

Unfortunately, POSIX systems do not have to interpret "-" on the
command line of crontab as a request to read from the standard
input.  Blindly adding "-" on the command line would not work as a
general solution.

Since POSIX tells us that cron must accept a file name argument, let's
solve this problem by specifying a temporary file instead.  This will
ensure that we work with the vast majority of implementations.

Note that because delete_tempfile closes the file for us, we should not
call fclose here on the handle, since doing so will introduce a double
free.

Reported-by: Renato Botelho <garga@FreeBSD.org>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-28 15:47:00 -07:00
Jeff King 969a564587 pack-bitmap-write: drop unused pack_idx_entry parameters
Our write_selected_commits_v1() function takes an array of
pack_idx_entry structs. We used to need them for computing commit
positions, but since aa30162559 (bitmap: move `get commit positions`
code to `bitmap_writer_finish`, 2022-08-14), the caller passes in a
separate array of positions for us. We can drop the unused array (and
its matching length parameter).

Likewise, when we added write_lookup_table() in 93eb41e240
(pack-bitmap-write.c: write lookup table extension, 2022-08-14), it
receives the same array of positions. So its "index" parameter was never
used at all.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-28 13:23:37 -07:00
René Scharfe c333c2ce65 test-mergesort: use mem_pool for sort input
The previous patch almost halved the number of heap allocations for the
sort subcommand.  Reduce it further by using a mem_pool for the line
objects.

Note that t/perf/run can't be used directly to compare two versions of
test-mergesort because it always runs the helpers from the checked-out
version.  So I hand-merged the results of separate runs before and with
this patch:

macOS 12.5.1 on M1:
0071.12: DEFINE_LIST_SORT unsorted     0.22(0.20+0.01)     0.21(0.19+0.01)
0071.14: DEFINE_LIST_SORT sorted       0.10(0.08+0.01)     0.10(0.08+0.01)
0071.16: DEFINE_LIST_SORT reversed     0.10(0.08+0.01)     0.10(0.08+0.01)

Git SDK 64-bit on Windows 11 21H2 on Ryzen 7 5800H:
0071.12: DEFINE_LIST_SORT unsorted     0.54(0.00+0.06)     0.44(0.01+0.06)
0071.14: DEFINE_LIST_SORT sorted       0.21(0.03+0.03)     0.19(0.04+0.01)
0071.16: DEFINE_LIST_SORT reversed     0.21(0.01+0.04)     0.19(0.04+0.04)

Debian bullseye on WSL2 on the same system:
0071.12: DEFINE_LIST_SORT unsorted     0.29(0.27+0.01)     0.22(0.19+0.02)
0071.14: DEFINE_LIST_SORT sorted       0.07(0.06+0.01)     0.06(0.04+0.02)
0071.16: DEFINE_LIST_SORT reversed     0.07(0.04+0.03)     0.06(0.04+0.02)

Signed-off-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-28 13:10:22 -07:00
René Scharfe f3e8ba2e64 test-mergesort: read sort input all at once
The sort subcommand of test-mergesort is used to test the performance of
sorting linked lists.  It reads lines from stdin, sorts them and prints
the result to stdout.  Two heap allocations are done per line: One for
the linked list item and one for the actual line string.  That imposes a
significant amount of allocation overhead.

Reduce it by doing the same as the sort subcommand of test-string-list,
namely to read the whole input file into a single buffer and then split
it in-place.

Note that t/perf/run can't be used directly to compare two versions of
test-mergesort because it always runs the helpers from the checked-out
version.  So I hand-merged the results of separate runs before and with
this patch:

macOS 12.5.1 on M1:
0071.12: DEFINE_LIST_SORT unsorted     0.23(0.20+0.01)     0.22(0.20+0.01)
0071.14: DEFINE_LIST_SORT sorted       0.12(0.10+0.01)     0.10(0.08+0.01)
0071.16: DEFINE_LIST_SORT reversed     0.12(0.10+0.01)     0.10(0.08+0.01)

Git SDK 64-bit on Windows 11 21H2 on Ryzen 7 5800H:
0071.12: DEFINE_LIST_SORT unsorted     0.71(0.00+0.03)     0.54(0.00+0.06)
0071.14: DEFINE_LIST_SORT sorted       0.42(0.00+0.04)     0.21(0.03+0.03)
0071.16: DEFINE_LIST_SORT reversed     0.42(0.06+0.01)     0.21(0.01+0.04)

Debian bullseye on WSL2 on the same system:
0071.12: DEFINE_LIST_SORT unsorted     0.41(0.39+0.02)     0.29(0.27+0.01)
0071.14: DEFINE_LIST_SORT sorted       0.11(0.08+0.02)     0.07(0.06+0.01)
0071.16: DEFINE_LIST_SORT reversed     0.11(0.08+0.02)     0.07(0.04+0.03)

Signed-off-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-28 13:10:20 -07:00
Eric Sunshine bcf325ae77 t4301: account for behavior differences between sed implementations
It is a common pattern in this script to write the result of
`merge-tree -z` (NUL-termination mode) to an "actual" file and then
manually append a newline to that file so that it can be diff'd easily
with a hand-crafted "expect" file which itself ends with a newline since
it has been created by standard Unix tools which terminate lines by
default. For instance:

    git merge-tree --write-tree -z ... >out &&
    printf "\\n" >>out
    anonymize_hash out >actual &&
    q_to_nul <<-EOF >expect &&
    ...
    EOF
    test_cmp expect actual

However, one test gets this backward:

    git merge-tree --write-tree -z ... >out &&
    anonymize_hash out >actual &&
    printf "\\n" >>actual

which means that, unlike all other cases, when anonymize_hash() is
called, the file being anonymized does not end with a newline. As a
result, this test fails on some platforms.

anonymize_hash() is implemented like this:

    anonymize_hash() {
        sed -e "s/[0-9a-f]\{40,\}/HASH/g" "$@"
    }

The problem arises due to differences in behavior of various `sed`
implementations when fed an incomplete line (lacking a newline).
Although most modern `sed` implementations output such a line
unmolested (i.e. without a newline), some older `sed` implementations
forcibly add a newline to the incomplete line (giving the output an
extra unexpected newline), while other very old implementations simply
swallow an incomplete line and don't emit it at all (making the output
shorter than expected).

Fix this test by manually adding the newline before passing it through
`sed`, thus ensuring identical behavior with all `sed` implementation,
and bringing the test in line with other tests in this script.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-28 13:09:46 -07:00
Junio C Hamano 2987ce743d Merge branch 'en/t4301-more-merge-tree-tests' into es/t4301-sed-portability-fix
* en/t4301-more-merge-tree-tests:
  t4301: add more interesting merge-tree testcases
2022-08-28 13:08:54 -07:00
René Scharfe babe2e0559 tempfile: avoid directory cleanup race
The temporary directory created by mks_tempfile_dt() is deleted by first
deleting the file within, then truncating the filename strbuf and
passing the resulting string to rmdir(2).  When the cleanup routine is
invoked concurrently by a signal handler we can end up passing the now
truncated string to unlink(2), however, which could cause problems on
some systems.

Avoid that issue by remembering the directory name separately.  This way
the paths stay unchanged.  A signal handler can still race with normal
cleanup, but deleting the same files and directories twice is harmless.

Reported-by: Jeff King <peff@peff.net>
Signed-off-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-27 10:17:46 -07:00
Junio C Hamano 07ee72db0e Sync with 'maint' 2022-08-26 11:14:11 -07:00
Junio C Hamano 0f5bd024f2 A handful more topics from the 'master' front for 2.37.3
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-26 11:13:13 -07:00
Junio C Hamano 842c912fc7 Merge branch 'po/doc-add-renormalize' into maint
Documentation for "git add --renormalize" has been improved.
source: <20220810144450.470-2-philipoakley@iee.email>

* po/doc-add-renormalize:
  doc add: renormalize is not idempotent for CRCRLF
2022-08-26 11:13:13 -07:00
Junio C Hamano 7be9f3f335 Merge branch 'vd/sparse-reset-checkout-fixes' into maint
Fixes to sparse index compatibility work for "reset" and "checkout"
commands.
source: <pull.1312.v3.git.1659985672.gitgitgadget@gmail.com>

* vd/sparse-reset-checkout-fixes:
  unpack-trees: unpack new trees as sparse directories
  cache.h: create 'index_name_pos_sparse()'
  oneway_diff: handle removed sparse directories
  checkout: fix nested sparse directory diff in sparse index
2022-08-26 11:13:13 -07:00
Junio C Hamano e5cb51d3aa Merge branch 'jk/fsck-tree-mode-bits-fix' into maint
"git fsck" reads mode from tree objects but canonicalizes the mode
before passing it to the logic to check object sanity, which has
hid broken tree objects from the checking logic.  This has been
corrected, but to help exiting projects with broken tree objects
that they cannot fix retroactively, the severity of anomalies this
code detects has been demoted to "info" for now.
source: <YvQcNpizy9uOZiAz@coredump.intra.peff.net>

* jk/fsck-tree-mode-bits-fix:
  fsck: downgrade tree badFilemode to "info"
  fsck: actually detect bad file modes in trees
  tree-walk: add a mechanism for getting non-canonicalized modes
2022-08-26 11:13:12 -07:00
Junio C Hamano 222f953777 Merge branch 'fc/vimdiff-layout-vimdiff3-fix' into maint
"vimdiff3" regression fix.
source: <20220810154618.307275-1-felipe.contreras@gmail.com>

* fc/vimdiff-layout-vimdiff3-fix:
  mergetools: vimdiff: simplify tabfirst
  mergetools: vimdiff: fix single window layouts
  mergetools: vimdiff: rework tab logic
  mergetools: vimdiff: fix for diffopt
  mergetools: vimdiff: silence annoying messages
  mergetools: vimdiff: make vimdiff3 actually work
  mergetools: vimdiff: fix comment
2022-08-26 11:13:12 -07:00
Junio C Hamano 10f9eab347 Merge branch 'js/safe-directory-plus' into maint
Platform-specific code that determines if a directory is OK to use
as a repository has been taught to report more details, especially
on Windows.
source: <pull.1286.v2.git.1659965270.gitgitgadget@gmail.com>

* js/safe-directory-plus:
  mingw: handle a file owned by the Administrators group correctly
  mingw: be more informative when ownership check fails on FAT32
  mingw: provide details about unsafe directories' ownership
  setup: prepare for more detailed "dubious ownership" messages
  setup: fix some formatting
2022-08-26 11:13:12 -07:00
Junio C Hamano 6283c1e6ad Merge branch 'pw/use-glibc-tunable-for-malloc-optim' into maint
Avoid repeatedly running getconf to ask libc version in the test
suite, and instead just as it once per script.
source: <pull.1311.git.1659620305757.gitgitgadget@gmail.com>

* pw/use-glibc-tunable-for-malloc-optim:
  tests: cache glibc version check
2022-08-26 11:13:12 -07:00
Junio C Hamano 5825304328 Merge branch 'ab/hooks-regression-fix' into maint
A follow-up fix to a fix for a regression in 2.36.
source: <patch-1.1-2450e3e65cf-20220805T141402Z-avarab@gmail.com>

* ab/hooks-regression-fix:
  hook API: don't segfault on strbuf_addf() to NULL "out"
2022-08-26 11:13:12 -07:00
Junio C Hamano ed051d4024 Merge branch 'gc/git-reflog-doc-markup' into maint
Doc mark-up fix.
source: <pull.1304.git.git.1659387885711.gitgitgadget@gmail.com>

* gc/git-reflog-doc-markup:
  Documentation/git-reflog: remove unneeded \ from \{
2022-08-26 11:13:11 -07:00
Junio C Hamano c2d62d0c7d Merge branch 'js/ort-clean-up-after-failed-merge' into maint
Plug memory leaks in the failure code path in the "merge-ort" merge
strategy backend.
source: <pull.1307.v2.git.1659114727.gitgitgadget@gmail.com>

* js/ort-clean-up-after-failed-merge:
  merge-ort: do leave trace2 region even if checkout fails
  merge-ort: clean up after failed merge
2022-08-26 11:13:11 -07:00
Junio C Hamano 4b2d41b0ad Merge branch 'jk/struct-zero-init-with-older-gcc' into maint
Older gcc with -Wall complains about the universal zero initializer
"struct s = { 0 };" idiom, which makes developers' lives
inconvenient (as -Werror is enabled by DEVELOPER=YesPlease).  The
build procedure has been tweaked to help these compilers.
source: <YuQ60ZUPBHAVETD7@coredump.intra.peff.net>

* jk/struct-zero-init-with-older-gcc:
  config.mak.dev: squelch -Wno-missing-braces for older gcc
2022-08-26 11:13:11 -07:00
Junio C Hamano 69c99b85e7 Merge branch 'js/lstat-mingw-enotdir-fix' into maint
Fix to lstat() emulation on Windows.
source: <pull.1291.v3.git.1659089152877.gitgitgadget@gmail.com>

* js/lstat-mingw-enotdir-fix:
  lstat(mingw): correctly detect ENOTDIR scenarios
2022-08-26 11:13:10 -07:00
Junio C Hamano 9166bca8ba Merge branch 'js/mingw-with-python' into maint
Conditionally allow building Python interpreter on Windows
source: <pull.1306.v2.git.1659109272.gitgitgadget@gmail.com>

* js/mingw-with-python:
  mingw: remove unneeded `NO_CURL` directive
  mingw: remove unneeded `NO_GETTEXT` directive
  windows: include the Python bits when building Git for Windows
2022-08-26 11:13:10 -07:00
Junio C Hamano 2794e813c6 Merge branch 'ca/unignore-local-installation-on-windows' into maint
Fix build procedure for Windows that uses CMake so that it can pick
up the shell interpreter from local installation location.
source: <pull.1304.git.1658912756815.gitgitgadget@gmail.com>

* ca/unignore-local-installation-on-windows:
  cmake: support local installations of git
2022-08-26 11:13:10 -07:00
Abhradeep Chakraborty 761416ef91 bitmap-lookup-table: add performance tests for lookup table
Add performance tests to verify the performance of lookup table.
`p5310-pack-bitmaps.sh` contain tests with and without lookup table.
`p5312-pack-bitmaps-revs.sh` contain same tests with and without
lookup table but with `pack.writeReverseIndex` enabled.

Lookup table makes Git run faster in most of the cases. Below is the
result of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`
gives similar result. The repository used in the test is linux kernel.

Test                                                    this tree
-----------------------------------------------------------------------
5310.4: enable lookup table: false                    0.01(0.00+0.00)
5310.5: repack to disk                                320.89(230.20+23.45)
5310.6: simulated clone                               14.04(5.78+1.79)
5310.7: simulated fetch                               1.95(3.05+0.20)
5310.8: pack to file (bitmap)                         44.73(20.55+7.45)
5310.9: rev-list (commits)                            0.78(0.46+0.10)
5310.10: rev-list (objects)                           4.07(3.97+0.08)
5310.11: rev-list with tag negated via --not          0.06(0.02+0.03)
         --all (objects)
5310.12: rev-list with negative tag (objects)         0.21(0.15+0.05)
5310.13: rev-list count with blob:none                0.24(0.17+0.06)
5310.14: rev-list count with blob:limit=1k            7.07(5.92+0.48)
5310.15: rev-list count with tree:0                   0.25(0.17+0.07)
5310.16: simulated partial clone                      5.67(3.28+0.64)
5310.18: clone (partial bitmap)                       16.05(8.34+1.86)
5310.19: pack to file (partial bitmap)                59.76(27.22+7.43)
5310.20: rev-list with tree filter (partial bitmap)   0.90(0.18+0.16)
5310.24: enable lookup table: true                    0.01(0.00+0.00)
5310.25: repack to disk                               319.73(229.30+23.01)
5310.26: simulated clone                              13.69(5.72+1.78)
5310.27: simulated fetch                              1.84(3.02+0.16)
5310.28: pack to file (bitmap)                        45.63(20.67+7.50)
5310.29: rev-list (commits)                           0.56(0.39+0.8)
5310.30: rev-list (objects)                           3.77(3.74+0.08)
5310.31: rev-list with tag negated via --not          0.05(0.02+0.03)
         --all (objects)
5310.32: rev-list with negative tag (objects)         0.21(0.15+0.05)
5310.33: rev-list count with blob:none                0.23(0.17+0.05)
5310.34: rev-list count with blob:limit=1k            6.65(5.72+0.40)
5310.35: rev-list count with tree:0                   0.23(0.16+0.06)
5310.36: simulated partial clone                      5.57(3.26+0.59)
5310.38: clone (partial bitmap)                       15.89(8.39+1.84)
5310.39: pack to file (partial bitmap)                58.32(27.55+7.47)
5310.40: rev-list with tree filter (partial bitmap)   0.73(0.18+0.15)

Test 4-15 are tested without using lookup table. Same tests are
repeated in 16-30 (using lookup table).

Mentored-by: Taylor Blau <me@ttaylorr.com>
Co-Mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Signed-off-by: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>
Reviewed-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-26 10:14:02 -07:00
Abhradeep Chakraborty 28cd730680 pack-bitmap: prepare to read lookup table extension
Earlier change teaches Git to write bitmap lookup table. But Git
does not know how to parse them.

Teach Git to parse the existing bitmap lookup table. The older
versions of Git are not affected by it. Those versions ignore the
lookup table.

Mentored-by: Taylor Blau <me@ttaylorr.com>
Co-Mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Signed-off-by: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>
Reviewed-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-26 10:13:58 -07:00
Abhradeep Chakraborty 76f14b777c pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests
Teach Git to provide a way for users to enable/disable bitmap lookup
table extension by providing a config option named 'writeBitmapLookupTable'.
Default is false.

Also add test to verify writting of lookup table.

Mentored-by: Taylor Blau <me@ttaylorr.com>
Co-Mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Co-Authored-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>
Reviewed-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-26 10:13:54 -07:00
Abhradeep Chakraborty 93eb41e240 pack-bitmap-write.c: write lookup table extension
The bitmap lookup table extension was documented by an earlier
change, but Git does not yet know how to write that extension.

Teach Git to write bitmap lookup table extension. The table contains
the list of `N` <commit_pos, offset, xor_row>` triplets. These
triplets are sorted according to their commit pos (ascending order).
The meaning of each data in the i'th triplet is given below:

  - commit_pos stores commit position (in the pack-index or midx).
    It is a 4 byte network byte order unsigned integer.

  - offset is the position (in the bitmap file) from which that
    commit's bitmap can be read.

  - xor_row is the position of the triplet in the lookup table
    whose bitmap is used to compress this bitmap, or `0xffffffff`
    if no such bitmap exists.

Mentored-by: Taylor Blau <me@ttaylorr.com>
Co-mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Co-authored-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>
Reviewed-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-26 10:13:50 -07:00
Abhradeep Chakraborty aa30162559 bitmap: move get commit positions code to bitmap_writer_finish
The `write_selected_commits_v1` function takes care of writing commit
positions along with their corresponding bitmaps in the disk. It is
OK because this `search commit position of a given commit` algorithm
is needed only once here. But in later changes of the `lookup table
extension series`, we need same commit positions which means we have
to run the above mentioned algorithm one more time.

Move the `search commit position of a given commit` algorithm to
`bitmap_writer_finish()` and use the `commit_positions` array
to get commit positions of their corresponding bitmaps.

Signed-off-by: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>
Reviewed-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-26 10:13:47 -07:00
Abhradeep Chakraborty e9977b12fd Documentation/technical: describe bitmap lookup table extension
When reading bitmap file, Git loads each and every bitmap one by one
even if all the bitmaps are not required. A "bitmap lookup table"
extension to the bitmap format can reduce the overhead of loading
bitmaps which stores a list of bitmapped commit id pos (in the midx
or pack, along with their offset and xor offset. This way Git can
load only the necessary bitmaps without loading the previous bitmaps.

Older versions of Git ignore the lookup table extension and don't
throw any kind of warning or error while parsing the bitmap file.

Add some information for the new "bitmap lookup table" extension in the
bitmap-format documentation.

Mentored-by: Taylor Blau <me@ttaylorr.com>
Co-Mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Co-Authored-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>
Reviewed-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-26 10:13:33 -07:00
Elijah Newren 6693fb3f01 t64xx: convert 'test_create_repo' to 'git init'
Convert the merge-specific tests (those in the t64xx range) over to
using 'git init' instead of 'test_create_repo'.

Reviewed-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-26 09:23:03 -07:00
Junio C Hamano 7c46ea0ded The fifteenth batch
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-25 14:42:32 -07:00
Junio C Hamano f00ddc9f48 Merge branch 'vd/scalar-generalize-diagnose'
The "diagnose" feature to create a zip archive for diagnostic
material has been lifted from "scalar" and made into a feature of
"git bugreport".

* vd/scalar-generalize-diagnose:
  scalar: update technical doc roadmap
  scalar-diagnose: use 'git diagnose --mode=all'
  builtin/bugreport.c: create '--diagnose' option
  builtin/diagnose.c: add '--mode' option
  builtin/diagnose.c: create 'git diagnose' builtin
  diagnose.c: add option to configure archive contents
  scalar-diagnose: move functionality to common location
  scalar-diagnose: move 'get_disk_info()' to 'compat/'
  scalar-diagnose: add directory to archiver more gently
  scalar-diagnose: avoid 32-bit overflow of size_t
  scalar-diagnose: use "$GIT_UNZIP" in test
2022-08-25 14:42:32 -07:00
Junio C Hamano a103ad6f3d Merge branch 'jk/pipe-command-nonblock'
Fix deadlocks between main Git process and subprocess spawned via
the pipe_command() API, that can kill "git add -p" that was
reimplemented in C recently.

* jk/pipe-command-nonblock:
  pipe_command(): mark stdin descriptor as non-blocking
  pipe_command(): handle ENOSPC when writing to a pipe
  pipe_command(): avoid xwrite() for writing to pipe
  git-compat-util: make MAX_IO_SIZE define globally available
  nonblock: support Windows
  compat: add function to enable nonblocking pipes
2022-08-25 14:42:32 -07:00
Junio C Hamano 098b7bfaa6 Merge branch 'js/fetch-negotiation-trace'
The common ancestor negotiation exchange during a "git fetch"
session now leaves trace log.

* js/fetch-negotiation-trace:
  fetch-pack: add tracing for negotiation rounds
2022-08-25 14:42:31 -07:00
Junio C Hamano 01a30a5a58 Merge branch 'jk/is-promisor-object-keep-tree-in-use'
An earlier optimization discarded a tree-object buffer that is
still in use, which has been corrected.

* jk/is-promisor-object-keep-tree-in-use:
  is_promisor_object(): fix use-after-free of tree buffer
2022-08-25 14:42:31 -07:00
Junio C Hamano df3c129e24 Merge branch 'en/submodule-merge-messages-fixes'
Further update the help messages given while merging submodules.

* en/submodule-merge-messages-fixes:
  merge-ort: provide helpful submodule update message when possible
  merge-ort: avoid surprise with new sub_flag variable
  merge-ort: remove translator lego in new "submodule conflict suggestion"
  submodule merge: update conflict error message
2022-08-25 14:42:29 -07:00
Jeff King 8f9d80f6c0 remote: run "remote rm" argv through parse_options()
The "git remote rm" command's option parsing is fairly primitive: it
insists on a single argument, which it treats as the remote name, and
displays a usage message otherwise.

This is OK, and maybe even convenient, as you could run:

  git remote rm --foo

to drop a remote named "--foo". But it's also weirdly unlike most of the
rest of Git, which would complain that there is no option "--foo". The
right way to spell it by our conventions is:

  git remote rm -- --foo

but this doesn't currently work.

So let's bring the command in line with the rest of Git (including its
sibling subcommands!) by feeding argv to parse_options(). We already
have an empty options array for the usage helper.

Note that we have to adjust the argc index down by one, as
parse_options() eats the program name from the start of the array.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-25 09:43:30 -07:00
Jeff King 0d330a53f3 maintenance: add parse-options boilerplate for subcommands
Several of the git-maintenance subcommands don't take any options, so
they don't bother looking at argv at all. This means they'll silently
accept garbage, like:

  $ git maintenance register --foo
  [no output]

  $ git maintenance stop bar
  [no output]

Let's give them the basic boilerplate to detect and handle these cases:

  $ git maintenance register --foo
  error: unknown option `foo'
  usage: git maintenance register

  $ git maintenance stop bar
  usage: git maintenance stop

We could reduce the number of lines of code here a bit with a shared
helper function. But it's worth building out the boilerplate, as it may
serve as the base for adding options later.

Note one complication: maintenance_start() calls directly into
maintenance_register(), so it now needs to pass a plausible argv (we
don't care, but parse_options() is expecting there to at least be an
argv[0] program name). This is an extra line of code, but it eliminates
the need for an explanatory comment.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-25 09:43:30 -07:00
Jeff King ecd2d3efe0 pass subcommand "prefix" arguments to parse_options()
Recent commits such as bf0a6b65fc (builtin/multi-pack-index.c: let
parse-options parse subcommands, 2022-08-19) converted a few functions
to match our usual argc/argv/prefix conventions, but the prefix argument
remains unused.

However, there is a good use for it: they should pass it to their own
parse_options() functions, where it may be used to adjust the value of
any filename options. In all but one of these functions, there's no
behavior change, since they don't use OPT_FILENAME. But this is an
actual fix for one option, which you can see by modifying the test suite
like so:

	diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh
	index 4fe57414c1..d0974d4371 100755
	--- a/t/t5326-multi-pack-bitmaps.sh
	+++ b/t/t5326-multi-pack-bitmaps.sh
	@@ -186,7 +186,11 @@ test_expect_success 'writing a bitmap with --refs-snapshot' '

	 		# Then again, but with a refs snapshot which only sees
	 		# refs/tags/one.
	-		git multi-pack-index write --bitmap --refs-snapshot=snapshot &&
	+		(
	+			mkdir subdir &&
	+			cd subdir &&
	+			git multi-pack-index write --bitmap --refs-snapshot=../snapshot
	+		) &&

	 		test_path_is_file $midx &&
	 		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&

I'd emphasize that this wasn't broken by bf0a6b65fc; it has been broken
all along, because the sub-function never got to see the prefix. It is
that commit which is actually enabling us to fix it (and which also
brought attention to the problem because it triggers -Wunused-parameter!)

The other functions changed here don't use OPT_FILENAME at all. In their
cases this isn't fixing anything visible, but it's following the usual
pattern and future-proofing them against somebody adding new options and
being surprised.

I didn't include a test for the one visible case above. We don't
generally test routine parse-options behavior for individual options.
The challenge here was finding the problem, and now that this has been
done, it's not likely to regress. Likewise, we could apply the patch
above to cover it "for free" but it makes reading the rest of the test
unnecessarily complicated.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-25 09:43:29 -07:00
Junio C Hamano 9905e80b0f t5329: notice a failure within a loop
We try to write "|| return 1" (or "|| exit 1" in a subshell) at the
end of a sequence of &&-chained command in a loop of our tests, so
that a failure of any step during the earlier iteration of the loop
can properly be caught.

There is one loop in this test script that is used to compute the
expected result, that will be later compared with an actual output
produced by the "test-tool pack-mtimes" command.  This particular
loop, however, is placed on the upstream side of a pipe, whose
non-zero exit code does not get noticed.

Emit a line that will never be produced by the "test-tool pack-mtimes"
to cause the later comparison to fail.  As we use test_cmp to compare
this "expected output" file with the "actual output", the "error
message" we are emitting into the expected output stream will stand
out and shown to the tester.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-24 14:24:17 -07:00
Derrick Stolee ef46584831 ci: update 'static-analysis' to Ubuntu 22.04
GitHub Actions scheduled a brownout of Ubuntu 18.04, which canceled all
runs of the 'static-analysis' job in our CI runs. Update to 22.04 to
avoid this as the brownout later turns into a complete deprecation.

The use of 18.04 was set in d051ed77ee (.github/workflows/main.yml: run
static-analysis on bionic, 2021-02-08) due to the lack of Coccinelle
being available on 20.04 (which continues today).

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-24 13:02:12 -07:00
Elijah Newren 3c4dbf556f t4301: add more interesting merge-tree testcases
This adds several tests of `merge-tree -z` extended conflict output
behavior to the testsuite, including some tests adapted from t6422.
These tests mark current behavior, not necessarily optimal behavior.  In
particular, some path_msg() calls might want to include additional
paths.

These testcases also make something clear about the <Conflicted file>
info section of the output.  That section consists of a sequence of
lines of the form
    <mode> <object> <stage> <filename>
where <stage> is always greater than 0 (since each line comes from a
conflicted file).  The lines correspond to conflicts that would be
placed in the index if we were doing a merge in a working tree.  It is
perhaps natural to assume that for any given line, the <object> and
<filename> correspond to a single <revision>:<filename> pair from one of
the commits being merged (or from the merge base).  This is true for
simple conflicts.  However, these testcases make it clear that this is
not the case in general.  For example, <object> may be the hash of a
three-way content merge of three different files (and with different
filenames).

The tests no longer pass under TEST_PASSES_SANITIZE_LEAK; it appears
that doing a directory rename with "git mv", among other possible
problems, triggers issues.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-24 09:42:36 -07:00
Elijah Newren ae15fd4116 merge: small code readability improvement
After our loop through the selected strategies, we compare best_strategy
to wt_strategy.  This is fine, but the fact that the code setting
best_strategy sets it to use_strategies[i]->name requires a little bit
of extra checking to determine that at the time of setting, that's the
same as wt_strategy.  Just setting best_strategy to wt_strategy makes it
a little easier to verify what the loop is doing, at least for this
reader.

Further, use_strategies[i]->name is used in a number of places, where we
could just use wt_strategy.  The latter takes less time for this reader
to parse (one variable name instead of three), so just use wt_strategy
to make the code slightly faster for human readers to parse.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-24 09:25:59 -07:00
Elijah Newren 5b1d30cabf merge: cleanup confusing logic for handling successful merges
builtin/merge.c has a loop over the specified strategies, where if they
all fail with conflicts, it picks the one with the least number of
conflicts.

In the codepath that finds a successful merge, if an automatic commit
was wanted, the code breaks out of the above loop, which makes sense.
However, if the user requested there be no automatic commit, the loop
would continue.  That seems weird; --no-commit should not affect the
choice of merge strategy, but the code as written makes one think it
does.  However, since the loop itself embeds "!merge_was_ok" as a
condition on continuing to loop, it actually would also exit early if
--no-commit was specified, it just exited from a different location.

Restructure the code slightly to make it clear that the loop will
immediately exit whenever we find a merge strategy that is successful.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-24 09:10:27 -07:00
Elijah Newren d3a9295ada merge: only apply autostash when appropriate
If a merge failed and we are leaving conflicts in the working directory
for the user to resolve, we should not attempt to apply any autostash.

Further, if we fail to apply the autostash (because either the merge
failed, or the user requested --no-commit), then we should instruct the
user how to apply it later.

Add a testcase verifying we have corrected this behavior.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-24 09:08:32 -07:00