mirror of
https://github.com/git/git
synced 2024-11-05 04:53:18 +00:00
ba5a81d52b
The murmur3 implementation in bloom.c has a bug when converting series
of 4 bytes into network-order integers when char is signed (which is
controllable by a compiler option, and the default signedness of char is
platform-specific). When a string contains characters with the high bit
set, this bug causes results that, although internally consistent within
Git, does not accord with other implementations of murmur3 (thus,
the changed path filters wouldn't be readable by other off-the-shelf
implementatios of murmur3) and even with Git binaries that were compiled
with different signedness of char. This bug affects both how Git writes
changed path filters to disk and how Git interprets changed path filters
on disk.
Therefore, introduce a new version (2) of changed path filters that
corrects this problem. The existing version (1) is still supported and
is still the default, but users should migrate away from it as soon
as possible.
Because this bug only manifests with characters that have the high bit
set, it may be possible that some (or all) commits in a given repo would
have the same changed path filter both before and after this fix is
applied. However, in order to determine whether this is the case, the
changed paths would first have to be computed, at which point it is not
much more expensive to just compute a new changed path filter.
So this patch does not include any mechanism to "salvage" changed path
filters from repositories. There is also no "mixed" mode - for each
invocation of Git, reading and writing changed path filters are done
with the same version number; this version number may be explicitly
stated (typically if the user knows which version they need) or
automatically determined from the version of the existing changed path
filters in the repository.
There is a change in write_commit_graph(). graph_read_bloom_data()
makes it possible for chunk_bloom_data to be non-NULL but
bloom_filter_settings to be NULL, which causes a segfault later on. I
produced such a segfault while developing this patch, but couldn't find
a way to reproduce it neither after this complete patch (or before),
but in any case it seemed like a good thing to include that might help
future patch authors.
The value in t0095 was obtained from another murmur3 implementation
using the following Go source code:
package main
import "fmt"
import "github.com/spaolacci/murmur3"
func main() {
fmt.Printf("%x\n", murmur3.Sum32([]byte("Hello world!")))
fmt.Printf("%x\n", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))
}
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
127 lines
3.3 KiB
Bash
Executable file
127 lines
3.3 KiB
Bash
Executable file
#!/bin/sh
|
|
|
|
test_description='Testing the various Bloom filter computations in bloom.c'
|
|
|
|
TEST_PASSES_SANITIZE_LEAK=true
|
|
. ./test-lib.sh
|
|
|
|
test_expect_success 'compute unseeded murmur3 hash for empty string' '
|
|
cat >expect <<-\EOF &&
|
|
Murmur3 Hash with seed=0:0x00000000
|
|
EOF
|
|
test-tool bloom get_murmur3 "" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute unseeded murmur3 hash for test string 1' '
|
|
cat >expect <<-\EOF &&
|
|
Murmur3 Hash with seed=0:0x627b0c2c
|
|
EOF
|
|
test-tool bloom get_murmur3 "Hello world!" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute unseeded murmur3 hash for test string 2' '
|
|
cat >expect <<-\EOF &&
|
|
Murmur3 Hash with seed=0:0x2e4ff723
|
|
EOF
|
|
test-tool bloom get_murmur3 "The quick brown fox jumps over the lazy dog" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute unseeded murmur3 hash for test string 3' '
|
|
cat >expect <<-\EOF &&
|
|
Murmur3 Hash with seed=0:0xa183ccfd
|
|
EOF
|
|
test-tool bloom get_murmur3_seven_highbit >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute bloom key for empty string' '
|
|
cat >expect <<-\EOF &&
|
|
Hashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|
|
|
Filter_Length:2
|
|
Filter_Data:11|11|
|
|
EOF
|
|
test-tool bloom generate_filter "" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute bloom key for whitespace' '
|
|
cat >expect <<-\EOF &&
|
|
Hashes:0xf178874c|0x5f3d6eb6|0xcd025620|0x3ac73d8a|0xa88c24f4|0x16510c5e|0x8415f3c8|
|
|
Filter_Length:2
|
|
Filter_Data:51|55|
|
|
EOF
|
|
test-tool bloom generate_filter " " >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute bloom key for test string 1' '
|
|
cat >expect <<-\EOF &&
|
|
Hashes:0xb270de9b|0x1bb6f26e|0x84fd0641|0xee431a14|0x57892de7|0xc0cf41ba|0x2a15558d|
|
|
Filter_Length:2
|
|
Filter_Data:92|6c|
|
|
EOF
|
|
test-tool bloom generate_filter "Hello world!" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute bloom key for test string 2' '
|
|
cat >expect <<-\EOF &&
|
|
Hashes:0x20ab385b|0xf5237fe2|0xc99bc769|0x9e140ef0|0x728c5677|0x47049dfe|0x1b7ce585|
|
|
Filter_Length:2
|
|
Filter_Data:a5|4a|
|
|
EOF
|
|
test-tool bloom generate_filter "file.txt" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success !SANITIZE_LEAK 'get bloom filters for commit with no changes' '
|
|
git init &&
|
|
git commit --allow-empty -m "c0" &&
|
|
cat >expect <<-\EOF &&
|
|
Filter_Length:1
|
|
Filter_Data:00|
|
|
EOF
|
|
test-tool bloom get_filter_for_commit "$(git rev-parse HEAD)" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'get bloom filter for commit with 10 changes' '
|
|
rm actual &&
|
|
rm expect &&
|
|
mkdir smallDir &&
|
|
for i in $(test_seq 0 9)
|
|
do
|
|
echo $i >smallDir/$i || return 1
|
|
done &&
|
|
git add smallDir &&
|
|
git commit -m "commit with 10 changes" &&
|
|
cat >expect <<-\EOF &&
|
|
Filter_Length:14
|
|
Filter_Data:02|b3|c4|a0|34|e7|fe|eb|cb|47|fe|a0|e8|72|
|
|
EOF
|
|
test-tool bloom get_filter_for_commit "$(git rev-parse HEAD)" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success EXPENSIVE 'get bloom filter for commit with 513 changes' '
|
|
rm actual &&
|
|
rm expect &&
|
|
mkdir bigDir &&
|
|
for i in $(test_seq 0 511)
|
|
do
|
|
echo $i >bigDir/$i || return 1
|
|
done &&
|
|
git add bigDir &&
|
|
git commit -m "commit with 513 changes" &&
|
|
cat >expect <<-\EOF &&
|
|
Filter_Length:1
|
|
Filter_Data:ff|
|
|
EOF
|
|
test-tool bloom get_filter_for_commit "$(git rev-parse HEAD)" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_done
|