git/reftable
Patrick Steinhardt ddac965965 reftable/writer: fix index corruption when writing multiple indices
Each reftable may contain multiple types of blocks for refs, objects and
reflog records, where each of these may have an index that makes it more
efficient to find the records. It was observed that the index for log
records can become corrupted under certain circumstances, where the
first entry of the index points into the object index instead of to the
log records.

As it turns out, this corruption can occur whenever we write a log index
as well as at least one additional index. Writing records and their index
is basically a two-step process:

  1. We write all blocks for the corresponding record. Each block that
     gets written is added to a list of blocks to index.

  2. Once all blocks were written we finish the section. If at least two
     blocks have been added to the list of blocks to index then we will
     now write the index for those blocks and flush it, as well.

When we have a very large number of blocks then we may decide to write a
multi-level index, which is why we also keep track of the list of the
index blocks in the same way as we previously kept track of the blocks
to index.

Now when we have finished writing all index blocks we clear the index
and flush the last block to disk. This is done in the wrong order though
because flushing the block to disk will re-add it to the list of blocks
to be indexed. The result is that the next section we are about to write
will have an entry in the list of blocks to index that points to the
last block of the preceding section's index, which will corrupt the log
index.

Fix this corruption by clearing the index after having written the last
block.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-01-03 09:54:20 -08:00
..
basics.c reftable: utility functions 2021-10-08 10:45:48 -07:00
basics.h reftable: utility functions 2021-10-08 10:45:48 -07:00
basics_test.c reftable: utility functions 2021-10-08 10:45:48 -07:00
block.c reftable/block: reuse buffer to compute record keys 2023-12-11 07:23:17 -08:00
block.h reftable/block: reuse buffer to compute record keys 2023-12-11 07:23:17 -08:00
block_test.c reftable/block: introduce macro to initialize struct block_iter 2023-12-11 07:23:17 -08:00
blocksource.c reftable: handle interrupted reads 2023-12-11 07:23:16 -08:00
blocksource.h reftable: add blocksource, an abstraction for random access reads 2021-10-08 10:45:48 -07:00
constants.h reftable: (de)serialization for the polymorphic record type. 2021-10-08 10:45:48 -07:00
dump.c hash-ll.h: split out of hash.h to remove dependency on repository.h 2023-04-24 12:47:32 -07:00
error.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
generic.c reftable: make assignments portable to AIX xlc v12.01 2022-03-28 13:58:10 -07:00
generic.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
iter.c reftable: make reftable_record a tagged union 2022-01-20 11:31:53 -08:00
iter.h reftable/block: introduce macro to initialize struct block_iter 2023-12-11 07:23:17 -08:00
LICENSE
merged.c reftable/merged: reuse buffer to compute record keys 2023-12-11 07:23:16 -08:00
merged.h reftable/merged: reuse buffer to compute record keys 2023-12-11 07:23:16 -08:00
merged_test.c reftable tests: avoid "int" overflow, use "uint64_t" 2022-01-13 13:39:09 -08:00
pq.c reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
pq.h reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
pq_test.c reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
publicbasics.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
reader.c reftable/block: introduce macro to initialize struct block_iter 2023-12-11 07:23:17 -08:00
reader.h reftable: read reftable files 2021-10-08 10:45:48 -07:00
readwrite_test.c reftable/writer: fix index corruption when writing multiple indices 2024-01-03 09:54:20 -08:00
record.c reftable: add print functions to the record types 2022-01-20 11:31:53 -08:00
record.h reftable: add print functions to the record types 2022-01-20 11:31:53 -08:00
record_test.c reftable: make assignments portable to AIX xlc v12.01 2022-03-28 13:58:10 -07:00
refname.c reftable: implement refname validation 2021-10-08 10:45:48 -07:00
refname.h reftable: implement refname validation 2021-10-08 10:45:48 -07:00
refname_test.c reftable: implement refname validation 2021-10-08 10:45:48 -07:00
reftable-blocksource.h reftable: add blocksource, an abstraction for random access reads 2021-10-08 10:45:48 -07:00
reftable-error.h reftable: signal overflow 2021-12-23 12:28:34 -08:00
reftable-generic.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
reftable-iterator.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
reftable-malloc.h reftable: utility functions 2021-10-08 10:45:48 -07:00
reftable-merged.h reftable: add merged table view 2021-10-08 10:45:48 -07:00
reftable-reader.h reftable: read reftable files 2021-10-08 10:45:48 -07:00
reftable-record.h reftable: make reftable-record.h function signatures const correct 2022-01-20 11:31:53 -08:00
reftable-stack.h reftable: implement stack, a mutable database of reftable files. 2021-10-08 10:45:48 -07:00
reftable-tests.h reftable: add a heap-based priority queue for reftable records 2021-10-08 10:45:48 -07:00
reftable-writer.h reftable: rename writer_stats to reftable_writer_stats 2022-02-23 13:36:26 -08:00
stack.c reftable/stack: do not auto-compact twice in reftable_stack_add() 2024-01-03 09:54:20 -08:00
stack.h reftable: implement stack, a mutable database of reftable files. 2021-10-08 10:45:48 -07:00
stack_test.c reftable/stack: perform auto-compaction with transactional interface 2023-12-11 07:23:16 -08:00
system.h hash-ll.h: split out of hash.h to remove dependency on repository.h 2023-04-24 12:47:32 -07:00
test_framework.c reftable: utility functions 2021-10-08 10:45:48 -07:00
test_framework.h reftable: wrap EXPECT macros in do/while 2023-12-11 07:23:15 -08:00
tree.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
tree.h reftable: a generic binary tree implementation 2021-10-08 10:45:48 -07:00
tree_test.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
writer.c reftable/writer: fix index corruption when writing multiple indices 2024-01-03 09:54:20 -08:00
writer.h reftable: write reftable files 2021-10-08 10:45:48 -07:00