Commit graph

12 commits

Author SHA1 Message Date
Junio C Hamano
6da9f888da Merge branch 'es/osx-header-pollutes-mask-macro'
* es/osx-header-pollutes-mask-macro:
  ewah: use less generic macro name
  ewah/bitmap: silence warning about MASK macro redefinition
2015-06-24 12:21:44 -07:00
Jeff King
34b935c01f ewah: use less generic macro name
The ewah/ewok.h header pollutes the global namespace with
"BITS_IN_WORD", without any specific notion that we are
talking about the bits in an eword_t. We can give this the
more specific name "BITS_IN_EWORD".

Signed-off-by: Jeff King <peff@peff.net>
Reviewed-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-06-03 00:04:01 -07:00
Eric Sunshine
414382fb00 ewah/bitmap: silence warning about MASK macro redefinition
On PowerPC Mac OS X (10.5.8 "Leopard" with Xcode 3.1),
system header /usr/include/ppc/param.h[1] pollutes the
preprocessor namespace with a macro generically named MASK.
This conflicts with the same-named macro in ewah/bitmap.c.
We can avoid this conflict by using a more specific name.

[1]: Included indirectly via:
     git-compat-util.h ->
     sys/sysctl.h ->
     sys/ucred.h ->
     sys/param.h ->
     machine/param.h ->
     ppc/param.h

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-06-03 00:03:03 -07:00
Nguyễn Thái Ngọc Duy
be0d9d5323 ewah: add convenient wrapper ewah_serialize_strbuf()
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-12 13:45:16 -07:00
Junio C Hamano
2c1f554d0c Merge branch 'jk/pack-bitmap'
The pack bitmap support did not build with older versions of GCC.

* jk/pack-bitmap:
  ewah: fix building with gcc < 3.4.0
2015-02-18 11:45:00 -08:00
Tom G. Christensen
bd4e8822da ewah: fix building with gcc < 3.4.0
The __builtin_ctzll function was added in gcc 3.4.0.
This extends the check for gcc so that use of __builtin_ctzll is only
enabled if gcc >= 3.4.0.

Signed-off-by: Tom G. Christensen <tgc@statsbiblioteket.dk>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-02-04 10:45:31 -08:00
Nguyễn Thái Ngọc Duy
16fc2b7a9c ewah: delete unused ewah_read_mmap_native declaration
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-04-29 12:43:16 -07:00
Nguyễn Thái Ngọc Duy
a0a2f7d79c ewah: fix constness of ewah_read_mmap
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-04-29 12:43:15 -07:00
Kyle J. McKay
68f4e1fc6a ewah_bitmap.c: do not assume size_t and eword_t are the same size
When buffer_grow changes the size of the buffer using realloc,
it first computes and saves the rlw pointer's offset into the
buffer using (uint8_t *) math before the realloc but then
restores it using (eword_t *) math.

In order to do this it's necessary to convert the (uint8_t *)
offset into an (eword_t *) offset.  It was doing this by
dividing by the sizeof(size_t).  Unfortunately sizeof(size_t)
is not same as sizeof(eword_t) on all platforms.

This causes illegal memory accesses and other bad things to
happen when attempting to use bitmaps on those platforms.

Fix this by dividing by the sizeof(eword_t) instead which
will always be correct for all platforms.

Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
Acked-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-04-22 16:21:16 -07:00
Jeff King
6b5b3a27b7 ewah: unconditionally ntohll ewah data
Commit a201c20 tried to optimize out a loop like:

  for (i = 0; i < len; i++)
	  data[i] = ntohll(data[i]);

in the big-endian case, because we know that ntohll is a
noop, and we do not need to pay the cost of the loop at all.
However, it mistakenly assumed that __BYTE_ORDER was always
defined, whereas it may not be on systems which do not
define it by default, and where we did not need to define it
to set up the ntohll macro. This includes OS X and Windows.

We could muck with the ordering in compat/bswap.h to make
sure it is defined unconditionally, but it is simpler to
still to just execute the loop unconditionally. That avoids
the application code knowing anything about these magic
macros, and lets it depend only on having ntohll defined.

And since the resulting loop looks like (on a big-endian
system):

  for (i = 0; i < len; i++)
	  data[i] = data[i];

any decent compiler can probably optimize it out.

Original report and analysis by Brian Gernhardt.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-02-12 11:21:29 -08:00
Vicent Marti
a201c20b41 ewah: support platforms that require aligned reads
The caller may hand us an unaligned buffer (e.g., because it
is an mmap of a file with many ewah bitmaps). On some
platforms (like SPARC) this can cause a bus error. We can
fix it with a combination of get_be32 and moving the data
into an aligned buffer (which we would do anyway, but we can
move it before fixing the endianness).

Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-01-23 14:05:05 -08:00
Vicent Marti
e1273106f6 ewah: compressed bitmap implementation
EWAH is a word-aligned compressed variant of a bitset (i.e. a data
structure that acts as a 0-indexed boolean array for many entries).

It uses a 64-bit run-length encoding (RLE) compression scheme,
trading some compression for better processing speed.

The goal of this word-aligned implementation is not to achieve
the best compression, but rather to improve query processing time.
As it stands right now, this EWAH implementation will always be more
efficient storage-wise than its uncompressed alternative.

EWAH arrays will be used as the on-disk format to store reachability
bitmaps for all objects in a repository while keeping reasonable sizes,
in the same way that JGit does.

This EWAH implementation is a mostly straightforward port of the
original `javaewah` library that JGit currently uses. The library is
self-contained and has been embedded whole (4 files) inside the `ewah`
folder to ease redistribution.

The library is re-licensed under the GPLv2 with the permission of Daniel
Lemire, the original author. The source code for the C version can
be found on GitHub:

	https://github.com/vmg/libewok

The original Java implementation can also be found on GitHub:

	https://github.com/lemire/javaewah

[jc: stripped debug-only code per Peff's $gmane/239768]

Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
Helped-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-12-30 12:17:20 -08:00