Travis: Reduce Toolchain cache item size by 73%

Empirically, every single push or PR has to download *and then upload*
about 3.6 GiB of "cache stuff", which takes up about 400 seconds:
https://travis-ci.com/github/SerenityOS/serenity/builds/177500795
On every single push/PR! No matter what!

Those 3.6 GB consist of:
- 3.2 GB Toolchain cache (around 260 MB per compressed item)
- 0.4 GB ccache, but is capped at 0.5 GB: https://travis-ci.com/github/BenWiederhake/serenity/builds/177528549
- (And 200 KB for some weird debian package? Dunno.)

Investigating in the size, the Toolchain consists mostly of *DEBUG SYMBOLS IN
THE COMPILER BINARIES* which comically misses the point. If we ever run into
compiler crashes, any stacktrace would be lost anyway as soon as the Travis VM
shuts down. Furthermore, Travis will only ever compile Serenity itself, and
Serenity forbids C in it's Contribution Guidelines. That's another 20 MB we
don't need to cache.

Stripping the binaries and deleting the C compiler reduces the uncompressed size
from 1200 MB down to 220 MB. The compressed size gets reduced from 260 MB to 70MB.
That's a reduction of 73%.

It'll take a while until the 'old' toolchains get deleted.
I guess it'll take less than a week.

From that point onward, the Travis cache will be 1.2 GB, consisting of:
- 0.7 GB Toolchain cache
- 0.5 GB ccache
- (And that weird 200 KB deb file)

If network speeds are linear, then this should reduce the "cache network
overhead time" from about 400 seconds to about 120 seconds.

tl;dr: Strip unnecessary debug infos, delete an unused files, and speed
everything up by two minutes. (Both Toolchain cache hits and Toolchain rebuilds!)
This commit is contained in:
Ben Wiederhake 2020-07-29 00:11:28 +02:00 committed by Andreas Kling
parent c6c9679cfc
commit 962e7855c5

View file

@ -19,9 +19,10 @@ MAKE="make"
MD5SUM="md5sum"
NPROC="nproc"
# Each cache entry is 260 MB. 8 entries are 4 GiB.
# It seems that Travis starts having trouble at 35 entries, so I think this is a good amount.
KEEP_CACHE_COUNT=8
# Each cache entry is 70 MB. 10 entries are 700 MiB.
# It seems that Travis starts having trouble around a total
# cache size of 9 GiB, so I think this is a good amount.
KEEP_CACHE_COUNT=10
if command -v ginstall &>/dev/null; then
INSTALL=ginstall
@ -110,7 +111,6 @@ pushd "$DIR"
ls -l
popd
fi
fi
popd
@ -262,7 +262,6 @@ popd
pushd "$DIR"
if [ "${TRY_USE_LOCAL_TOOLCHAIN}" = "y" ] ; then
# TODO: Compress with -z. It's factor 3, and costs no time.
echo "Caching toolchain:"
if [ -z "${DEPS_HASH}" ] ; then
@ -275,6 +274,23 @@ pushd "$DIR"
echo "Not touching cache then."
else
mkdir -p Cache/
# We *most definitely* don't need debug symbols in the linker/compiler.
# This cuts the uncompressed size from 1.2 GiB per Toolchain down to about 250 MiB.
pushd "Local/libexec/gcc/i686-pc-serenity/${GCC_VERSION}"
for binary in cc1plus lto1; do
echo "Before: $(du -h "${binary}")"
strip "${binary}"
echo "After: $(du -h "${binary}")"
done
# C is forbidden anyway by the Contribution Guidelines, so we can delete the C compiler:
echo "Before: $(du -h "cc1")"
rm cc1
echo "After: 0 cc1"
popd
binary=Local/bin/i686-pc-serenity-lto-dump
echo "Before: $(du -h "${binary}")"
strip "${binary}"
echo "After: $(du -h "${binary}")"
tar czf "Cache/ToolchainLocal_${DEPS_HASH}.tar.gz" Local/
fi
fi