git/reftable
Patrick Steinhardt e7485601ca reftable/writer: fix writing multi-level indices
When finishing a section we will potentially write an index that makes
it more efficient to look up relevant blocks. The index records written
will encode, for each block of the indexed section, what the offset of
that block is as well as the last key of that block. Thus, the reader
would iterate through the index records to find the first key larger or
equal to the wanted key and then use the encoded offset to look up the
desired block.

When there are a lot of blocks to index though we may end up writing
multiple index blocks, too. To not require a linear search across all
index blocks we instead end up writing a multi-level index. Instead of
referring to the block we are after, an index record may point to
another index block. The reader will then access the highest-level index
and follow down the chain of index blocks until it hits the sought-after
block.

It has been observed though that it is impossible to seek ref records of
the last ref block when using a multi-level index. While the multi-level
index exists and looks fine for most of the part, the highest-level
index was missing an index record pointing to the last block of the next
index. Thus, every additional level made more refs become unseekable at
the end of the ref section.

The root cause is that we are not flushing the last block of the current
level once done writing the level. Consequently, it wasn't recorded in
the blocks that need to be indexed by the next-higher level and thus we
forgot about it.

Fix this bug by flushing blocks after we have written all index records.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-02-01 11:11:32 -08:00
..
basics.c reftable: utility functions 2021-10-08 10:45:48 -07:00
basics.h reftable: utility functions 2021-10-08 10:45:48 -07:00
basics_test.c reftable: utility functions 2021-10-08 10:45:48 -07:00
block.c reftable/block: reuse buffer to compute record keys 2023-12-11 07:23:17 -08:00
block.h reftable/block: reuse buffer to compute record keys 2023-12-11 07:23:17 -08:00
block_test.c reftable/record: store "val1" hashes as static arrays 2024-01-03 09:54:20 -08:00
blocksource.c reftable: handle interrupted reads 2023-12-11 07:23:16 -08:00
blocksource.h reftable: add blocksource, an abstraction for random access reads 2021-10-08 10:45:48 -07:00
constants.h reftable: (de)serialization for the polymorphic record type. 2021-10-08 10:45:48 -07:00
dump.c treewide: remove unnecessary includes in source files 2023-12-26 12:04:31 -08:00
error.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
generic.c treewide: remove unnecessary includes in source files 2023-12-26 12:04:31 -08:00
generic.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
iter.c reftable: make reftable_record a tagged union 2022-01-20 11:31:53 -08:00
iter.h reftable/block: introduce macro to initialize struct block_iter 2023-12-11 07:23:17 -08:00
LICENSE
merged.c Merge branch 'ps/reftable-fixes-and-optims' 2024-01-16 10:11:57 -08:00
merged.h reftable/merged: reuse buffer to compute record keys 2023-12-11 07:23:16 -08:00
merged_test.c reftable: honor core.fsync 2024-01-23 13:45:27 -08:00
pq.c reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
pq.h reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
pq_test.c reftable: use a pointer for pq_entry param 2022-09-15 11:32:37 -07:00
publicbasics.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
reader.c reftable/reader: be more careful about errors in indexed seeks 2024-02-01 11:11:32 -08:00
reader.h reftable: read reftable files 2021-10-08 10:45:48 -07:00
readwrite_test.c reftable/writer: fix writing multi-level indices 2024-02-01 11:11:32 -08:00
record.c reftable/record: store "val2" hashes as static arrays 2024-01-03 09:54:21 -08:00
record.h reftable: add print functions to the record types 2022-01-20 11:31:53 -08:00
record_test.c reftable/record: store "val2" hashes as static arrays 2024-01-03 09:54:21 -08:00
refname.c reftable: implement refname validation 2021-10-08 10:45:48 -07:00
refname.h reftable: implement refname validation 2021-10-08 10:45:48 -07:00
refname_test.c reftable: honor core.fsync 2024-01-23 13:45:27 -08:00
reftable-blocksource.h reftable: add blocksource, an abstraction for random access reads 2021-10-08 10:45:48 -07:00
reftable-error.h reftable: signal overflow 2021-12-23 12:28:34 -08:00
reftable-generic.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
reftable-iterator.h reftable: generic interface to tables 2021-10-08 10:45:48 -07:00
reftable-malloc.h reftable: utility functions 2021-10-08 10:45:48 -07:00
reftable-merged.h reftable: add merged table view 2021-10-08 10:45:48 -07:00
reftable-reader.h reftable: read reftable files 2021-10-08 10:45:48 -07:00
reftable-record.h reftable/record: store "val2" hashes as static arrays 2024-01-03 09:54:21 -08:00
reftable-stack.h reftable: implement stack, a mutable database of reftable files. 2021-10-08 10:45:48 -07:00
reftable-tests.h reftable: add a heap-based priority queue for reftable records 2021-10-08 10:45:48 -07:00
reftable-writer.h reftable: honor core.fsync 2024-01-23 13:45:27 -08:00
stack.c reftable/stack: fsync "tables.list" during compaction 2024-01-30 11:56:15 -08:00
stack.h reftable: implement stack, a mutable database of reftable files. 2021-10-08 10:45:48 -07:00
stack_test.c Merge branch 'ps/reftable-fixes-and-optims' 2024-01-16 10:11:57 -08:00
system.h hash-ll.h: split out of hash.h to remove dependency on repository.h 2023-04-24 12:47:32 -07:00
test_framework.c reftable: honor core.fsync 2024-01-23 13:45:27 -08:00
test_framework.h reftable: honor core.fsync 2024-01-23 13:45:27 -08:00
tree.c reftable: ensure git-compat-util.h is the first (indirect) include 2023-04-24 12:47:33 -07:00
tree.h reftable: a generic binary tree implementation 2021-10-08 10:45:48 -07:00
tree_test.c treewide: remove unnecessary includes in source files 2023-12-26 12:04:31 -08:00
writer.c reftable/writer: fix writing multi-level indices 2024-02-01 11:11:32 -08:00
writer.h reftable: honor core.fsync 2024-01-23 13:45:27 -08:00