This will still build the compiler for the target but will not build the
bootstrap cross-compiler in the cross-tools phase. Other toolchain
bootstrapping, such as elftoolchan and binutils, currently still occurs.
This will utilize the default CC (cc, /usr/bin/cc) as an external compiler.
This is planned to be on-by-default eventually.
This will utilize the __FreeBSD_cc_version compiler macro defined in the
source tree and compare it to CC's version. If they match then the
cross-compiler is skipped. If [X]CC is an external compiler (absolute
path) or WITHOUT_CROSS_COMPILER is already set, then this logic is skipped.
If the expected bootstrap compiler type no longer matches the found CC
compiler type (clang vs gcc), then the logic is skipped. As an extra
safety check the version number is also compared from the compiler to
the tree version.
Clang:
The macro FREEBSD_CC_VERSION is defined in:
lib/clang/include/clang/Basic/Version.inc
For clang -target will be used if TARGET_ARCH != MACHINE_ARCH. This
is from the current external toolchain logic. There is currently an
assumption that the host compiler can build the TARGET_ARCH. This
will usually be the case since we don't conditionalize target arch
support in clang, but it will break when introducing new
architectures. This problem is mitigated by incrementing the version
when adding new architectures.
GCC:
The macro FBSD_CC_VER is defined in:
gnu/usr.bin/cc/cc_tools/freebsd-native.h
For GCC there is no simple -target support when TARGET_ARCH !=
MACHINE_ARCH. In this case the opportunistic skip is not done. If we
add proper support for this case in external toolchain logic then it
will be fine to enable.
This relies on the macros being incremented whenever any change occurs
to these compilers that warrant rebuilding files. It also should never
repeat earlier values.
Reviewed by: brooks, bapt, imp
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D6357
`BEFORE: netif` was already in etc/rc.d/atm1, so no additional changes
are needed in that script
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
In case ELF Tool Chain's elfcopy introduces any regressions this knob is
available as a transition aid. It will be removed once we are confident
that any regressions have been fixed.
Tools removed from the list are now provided by the ELF Tool Chain
project and are not controlled by the WITHOUT_BINUTILS_BOOTSTRAP knob.
Sponsored by: The FreeBSD Foundation
This is needed to be able to run check-links.sh against a "sysrooted"
binary while ensuring that the ldd(1) call done on the host uses the
host libc. It is not possible to set LD_LIBRARY_PATH before calling
check-links.sh as then the "sysrooted" libc would be incorrectly used.
A LD_PRELOAD=libc.so is used to ldd(1) as it needs to use the host libc
to run. ldd(1) is a simple wrapper around execve(2) and dlopen(2) with
env LD_TRACE_LOADED_OBJECTS set. Due to the dlopen(2) restriction on
shared library tracing ldd(1) is still required for this lookup.
Sponsored by: EMC / Isilon Storage Division
It is built in libgcc_s.so and libgcc_eh.a to simplify transition.
It is enabled by default on arm64 (where we previously had no other
unwinder) and may be enabled for testing on other platforms by setting
WITH_LLVM_LIBUNWIND in src.conf(5).
Also add compiler-rt's __gcc_personality_v0 implementation for use with
the LLVM unwinder.
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D4787
bugfix-only release, with no new features.
Please note that from 3.5.0 onwards, clang and llvm require C++11
support to build; see UPDATING for more information.
It may only be used with WITH_AUTO_OBJ, which the WITH_DIRDEPS_BUILD does. We
could support this in the normal build as well if we forced creating the directory
and setting .OBJDIR.
Sponsored by: EMC / Isilon Storage Division
This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.
Differential Revision: https://reviews.freebsd.org/D4153
Reviewed by: bdrewery
- Move all section 5 bluetooth manpages under MK_BLUETOOTH != no
MFC after: 3 days
PR: 193260
Reported by: Philippe Michel <philippe.michel7@sfr.fr>
Sponsored by: EMC / Isilon Storage Division
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build. This is explained in
more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times. When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'. Stats are provided at the end of this message.
This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.
The option name was picked to match the existing option in ports. This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.
Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead. ccache does nothing special in these modes, although there is
no harm in calling it for them.
CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss. For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct. Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.
The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it. Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings. GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.
Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual. Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization. There is also a memcached backend available [5]. Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout. A
possible working layout is as follows:
Source: /some/prefix/src1
Source: /some/prefix/src2
Source: /some/prefix/src3
Objdir: /some/prefix/obj
Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative. Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.
distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.
The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).
Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files and ccache objects.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.
buildworld:
x buildworld-before
+ buildworld-ccache-empty
* buildworld-ccache-full
% buildworld-ccache-full-fastdep
# buildworld-fastdep
+-------------------------------------------------------------------------------+
|% * # +|
|% * # +|
|% * # xxx +|
| |A |
| A|
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 4519 4525.04 4520.73 4521.59 3.1104823
Difference at 95.0% confidence
758.027 +/- 43.4565
20.1412% +/- 1.15466%
(Student's t, pooled s = 19.1726)
* 3 1823.08 1827.2 1825.62 1825.3 2.0785572
Difference at 95.0% confidence
-1938.26 +/- 43.298
-51.5007% +/- 1.15045%
(Student's t, pooled s = 19.1026)
% 3 1266.96 1279.37 1270.47 1272.2667 6.3971113
Difference at 95.0% confidence
-2491.3 +/- 44.3704
-66.1952% +/- 1.17895%
(Student's t, pooled s = 19.5758)
# 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-ccache-empty
* buildkernel-ccache-empty-fastdep
% buildkernel-ccache-full
# buildkernel-ccache-full-fastdep
@ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|# @ % * |
|# @ % * x + |
|# @ % * xx ++|
| MA |
| MA|
| A |
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 727.97 731.91 728.06 729.31333 2.2492295
Difference at 95.0% confidence
156.88 +/- 4.17129
27.4058% +/- 0.728695%
(Student's t, pooled s = 1.84034)
* 3 527.1 528.29 528.08 527.82333 0.63516402
Difference at 95.0% confidence
-44.61 +/- 2.33254
-7.79305% +/- 0.407478%
(Student's t, pooled s = 1.02909)
% 3 400.4 401.05 400.62 400.69 0.3306055
Difference at 95.0% confidence
-171.743 +/- 2.16453
-30.0023% +/- 0.378128%
(Student's t, pooled s = 0.954969)
# 3 201.94 203.34 202.28 202.52 0.73020545
Difference at 95.0% confidence
-369.913 +/- 2.40293
-64.6212% +/- 0.419774%
(Student's t, pooled s = 1.06015)
@ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30
PR: 182944 [4]
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes
This speeds up buildworld by 16% on my system and buildkernel by 35%.
Rather than calling mkdep(1), which is just a wrapper around 'cc -E',
use the modern -MD -MT -MF flags to gather and generate dependencies during
compilation. This flag was introduced in GCC "a long time ago", in GCC 3.0,
and is also supported by Clang. (It appears that ICC also supports this but I
do not have access to test it). This avoids running the preprocessor *twice*
for every build, in both 'make depend' and 'make all'. This is especially
noticeable when using ccache since it does not cache preprocessor results from
mkdep(1) / 'cc -E', but still speeds up compilation with the -MD flags.
For 'make depend' a tree-walk is still done to ensure that all DPSRCS
are generated when expected, and that beforedepend/afterdepend and
_EXTRADEPEND are all still respected. In time this may change but for now
I've been conservative. The time for a tree-walk with -j combined with
SUBDIR_PARALLEL is not significant. For example, it takes about 9 seconds
with -j15 to walk all of src/ for 'make depend' now on my system.
A .depend file is still generated with the various rules that apply to
the final target, or custom rules. Otherwise there are now
per-built-object-file .depend files, such as .depend.filename.o. These
are included directly by make rather than populating .depend with a loop
and .depend lines, which only added overhead to the now almost-NOP 'make
depend' phase.
Before this I experimented with having mkdep(1) called in parallel per-file.
While this improved the kernel and lib/libc 'make depend' phase, it resulted
in slower build times overall.
The -M flags are removed from CFLAGS when linking since they have no effect.
Enabling this by default, for src or out-of-src, can be done once more testing
has been done, such as a ports exp-run, and with more compilers.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_FAST_DEPEND=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
buildworld:
x buildworld-before
+ buildworld-fastdep
+-------------------------------------------------------------------------------+
|+ |
|+ |
|+ xx x|
| |_MA___||
|A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|+ x |
|++ xx|
| A||
|A| |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
Sponsored by: EMC / Isilon Storage Division
MFC after: 3 weeks
Relnotes: yes
The command was checking local/remote system uptime, so rename the script to
match its function and to avoid confusion
The controlling variable in /etc/periodic.conf has been renamed from
daily_status_rwho_enable to daily_status_uptime_enable.
MFC after: 3 days
Reported by: Peter Jeremy <peter@rulingia.com>
Relnotes: yes
Sponsored by: EMC / Isilon Storage Division
MK_USB != no
Add the manpages to OptionalObsoleteFiles.inc
As a side-effect, this also fixes installworld with MK_USB == no
X-MFC with: r290128
Sponsored by: EMC / Isilon Storage Division
Extend OptionalObsoleteFiles.inc to delete all lib32 files when MK_LIB32 is
set to no on a system that previously had lib32 libraries installed.
Also, to prevent "make delete-old-dirs" from always deleting lib32 directories
after an installworld, move the lib32 subtree to its own mtree file that only
gets applied when MK_LIB32=yes.
Test: Ran "make delete-old" and "make delete-old-libs" on a system that never
had MK_LIB32 enabled, and on a system where MK_LIB32 was enabled and later
disabled. Did this both on amd64 and powerpc64.
Test: Ran "make tinderbox" without errors.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D3923
Also note that these env-only vars can be specified on the command line.
This fixes the dependent options that are env-only (such as WITH_META_MODE
and WITH_AUTO_OBJ) to properly display their dependencies.
Sponsored by: EMC / Isilon Storage Division
The option was added only to ease the transition from GNU Binutils to
ELF Tool Chain tools, and that process is now complete (for the viable
replacements). Noting the removal in UPDATING is sufficient as we have
not shipped a release with the option.
Reviewed by: brooks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3240
