git/reftable
Patrick Steinhardt 5c086453ff reftable/stack: perform auto-compaction with transactional interface
Whenever updating references or reflog entries in the reftable stack, we
need to add a new table to the stack, thus growing the stack's length by
one. The stack can grow to become quite long rather quickly, leading to
performance issues when trying to read records. But besides performance
issues, this can also lead to exhaustion of file descriptors very
rapidly as every single table requires a separate descriptor when
opening the stack.

While git-pack-refs(1) fixes this issue for us by merging the tables, it
runs too irregularly to keep the length of the stack within reasonable
limits. This is why the reftable stack has an auto-compaction mechanism:
`reftable_stack_add()` will call `reftable_stack_auto_compact()` after
its added the new table, which will auto-compact the stack as required.

But while this logic works alright for `reftable_stack_add()`, we do not
do the same in `reftable_addition_commit()`, which is the transactional
equivalent to the former function that allows us to write multiple
updates to the stack atomically. Consequentially, we will easily run
into file descriptor exhaustion in code paths that use many separate
transactions like e.g. non-atomic fetches.

Fix this issue by calling `reftable_stack_auto_compact()`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-12-11 07:23:16 -08:00
..
basics.c reftable: utility functions 2021-10-08 10:45:48 -07:00
basics.h reftable: utility functions 2021-10-08 10:45:48 -07:00
basics_test.c reftable: utility functions 2021-10-08 10:45:48 -07:00
block.c reftable: avoid writing empty keys at the block layer 2022-02-23 13:36:26 -08:00
block.h reftable: fix typo in header 2021-12-23 12:28:28 -08:00
block_test.c reftable: avoid writing empty keys at the block layer 2022-02-23 13:36:26 -08:00
blocksource.c reftable: handle interrupted reads 2023-12-11 07:23:16 -08:00
blocksource.h reftable: add blocksource, an abstraction for random access reads 2021-10-08 10:45:48 -07:00
constants.h reftable: (de)serialization for the polymorphic record type. 2021-10-08 10:45:48 -07:00
dump.c hash-ll.h: split out of hash.h to remove dependency on repository.h 2023-04-24 12:47:32 -07:00
error.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
generic.c reftable: make assignments portable to AIX xlc v12.01 2022-03-28 13:58:10 -07:00
generic.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
iter.c reftable: make reftable_record a tagged union 2022-01-20 11:31:53 -08:00
iter.h reftable: read reftable files 2021-10-08 10:45:48 -07:00
LICENSE
merged.c reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
merged.h reftable: add merged table view 2021-10-08 10:45:48 -07:00
merged_test.c reftable tests: avoid "int" overflow, use "uint64_t" 2022-01-13 13:39:09 -08:00
pq.c reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
pq.h reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
pq_test.c reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
publicbasics.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
reader.c reftable: drop unused parameter from reader_seek_linear() 2022-08-20 14:14:55 -07:00
reader.h reftable: read reftable files 2021-10-08 10:45:48 -07:00
readwrite_test.c reftable: rename writer_stats to reftable_writer_stats 2022-02-23 13:36:26 -08:00
record.c reftable: add print functions to the record types 2022-01-20 11:31:53 -08:00
record.h reftable: add print functions to the record types 2022-01-20 11:31:53 -08:00
record_test.c reftable: make assignments portable to AIX xlc v12.01 2022-03-28 13:58:10 -07:00
refname.c reftable: implement refname validation 2021-10-08 10:45:48 -07:00
refname.h reftable: implement refname validation 2021-10-08 10:45:48 -07:00
refname_test.c reftable: implement refname validation 2021-10-08 10:45:48 -07:00
reftable-blocksource.h reftable: add blocksource, an abstraction for random access reads 2021-10-08 10:45:48 -07:00
reftable-error.h reftable: signal overflow 2021-12-23 12:28:34 -08:00
reftable-generic.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
reftable-iterator.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
reftable-malloc.h reftable: utility functions 2021-10-08 10:45:48 -07:00
reftable-merged.h reftable: add merged table view 2021-10-08 10:45:48 -07:00
reftable-reader.h reftable: read reftable files 2021-10-08 10:45:48 -07:00
reftable-record.h reftable: make reftable-record.h function signatures const correct 2022-01-20 11:31:53 -08:00
reftable-stack.h reftable: implement stack, a mutable database of reftable files. 2021-10-08 10:45:48 -07:00
reftable-tests.h reftable: add a heap-based priority queue for reftable records 2021-10-08 10:45:48 -07:00
reftable-writer.h reftable: rename writer_stats to reftable_writer_stats 2022-02-23 13:36:26 -08:00
stack.c reftable/stack: perform auto-compaction with transactional interface 2023-12-11 07:23:16 -08:00
stack.h reftable: implement stack, a mutable database of reftable files. 2021-10-08 10:45:48 -07:00
stack_test.c reftable/stack: perform auto-compaction with transactional interface 2023-12-11 07:23:16 -08:00
system.h hash-ll.h: split out of hash.h to remove dependency on repository.h 2023-04-24 12:47:32 -07:00
test_framework.c reftable: utility functions 2021-10-08 10:45:48 -07:00
test_framework.h reftable: wrap EXPECT macros in do/while 2023-12-11 07:23:15 -08:00
tree.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
tree.h reftable: a generic binary tree implementation 2021-10-08 10:45:48 -07:00
tree_test.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
writer.c Merge branch 'ep/maint-equals-null-cocci' 2022-05-20 15:26:59 -07:00
writer.h reftable: write reftable files 2021-10-08 10:45:48 -07:00