Now that the minor issues preventing zfs.c from using CFLAGS_EARLY have
been fixed, use that mechanism like everything else that needs the
OpenZFS spl headers. This simplifies things somewhat. Update comments to
document why zfs.c is still special, though in different ways.
Note: We also use the fact that NEED_SOLARIS_BOOLEAN is only defined in
an environment where the solaris compat boolean stuff will be defined
prior to this point (eg, when we're building zfs.c in libsa), but not in
other environments (like when we're building mkimage and stand-alone
boot loaders that don't use libsa). These latter uses should be changed
to use the same ZFS compile env, but aren't as part of this commit.
This has to be done in the same change as the ZFS_EARLY change to not
break zfs.c building for one commit affecting bisectabiltiy.
Sponsored by: Netflix
Reviewed by: tsoome, delphij
Differential Revision: https://reviews.freebsd.org/D35894
Constrain CFLAGS for ZFS: don't add anything globally. Add the includes
to only the files that need them. Add -DHAS_ZSTD_ZFS to zfs.c (which
includes zfsimpl.c which includes zfssubr.c both of which need this
defined). Also add it to efi/boot1/Makefile since zfs_module.c also
includes zfsimple.c.
Sponsored by: Netflix
Reviewed by: tsoome
Differential Revision: https://reviews.freebsd.org/D35887
Various GELI sources need bootstrap.h and disk.h. In theory they
shouldn't need anything outside of libsa, but disk.h and bootstrap.h are
currently required.
This fixes the build with MK_LOADER_ZFS=no.
Obtained from: CheriBSD
Fixes: eaf7aabddc stand: geli CFLAGS tightening
Sponsored by: DARPA
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D35861
In theory they shouldn't need anything outside of libsa, but disk.h and
bootstrap.h are currently required. Future work wil address this issue.
This fixes the build with MK_LOADER_ZFS=no. ZFS's Makefile.inc adds
these flags globally to CFLAGS when it should not. This masked the
problem because the tools/boot/universe.sh didn't build MK_LOADER_ZFS=no
as part of its regressions. Future work will also fix this.
Obtained from: CheriBSD
Fixes: 84bf2bbbec stand: constrain zlib/gzip CFLAGS better
Sponsored by: DARPA
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D35860
Add comments describing the weird nesting things we have to do to live
inside the ZFS world. Also fix a stale comment by moving nvlist.c to an
early user.
Sponsored by: Netflix
Add the necessary glue to get blake3 building for the boot loaded as
well as connected to the ZFS system so it is useful.
On some platforms, we create references to blake3_sse2_impl and
blake3_sse41_impl ops structs to utilize SIMD. These aren't present on
x86 (since we dind't ask for them), but are on aarch64 with no
implementation. Since we don't want SIMD in the boot loader, have these
all return 'unsupported' always. This should be fixed upstream to allow
more flexibility in this selection, but for now we use this hack to not
modify the sys/contrib/openzfs with difficult to maintain hacks while
an upstreamable solution is found.
tsoome@ did the implementation bits in sys/cddl/boot, and I did the
Makefile work and the aweful blake3_impl_hack.c.
Co-author: tsoome@freebsd.org
Sponsored by: Netflix
Reviewed by: kevans (earlier version)
Differential Revision: https://reviews.freebsd.org/D35750
We no longer need to use this to get zfs_zstd.c compiling, so delete it.
Sponsored by: Netflix
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D35750
Now that we have the ability to work around all the issues that
zstd_stub.c worked around, compile zfs_std.c directly.
Sponsored by: Netflix
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D35750
sys/blake3.h is not safe to include in a standalone environment. It
assumes, unwisely, that there's only kernel or userland. The userland
choice is bad due to imperfections in how we handle stdlib.h in this
environment (we wind up including the host's stdlib.h, it expects a
standard FreeBSD setup which the gymnastics we've done to create a
OpenSolaris/Illumos-like environment interfere with). Thankfully, in
this case we can just pretend to be the kernel for a little bit by
defining _KERNEL and the undefing it after we exit.
Sponsored by: Netflix
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D35750
There's a number of issues with including zfs_context.h from a
standalone environment. First, sys/uio_imp.h isn't at all safe for this
environment, so define its guard #defines so that its contents are
skipped. Next, there's a problem including string.h to get the mem*
routines, so just define them here. ZFS_MODULE_PARAM_ARGS isn't defined
properly. I had wanted to define it when I was upstreaming changes to
include/os/freebsd/spl/sys/zfs_context.h, but they ran into resistance
so I'm defining that here now (it is also defined in zstd_shim.c, but
that will disappear once the issues it works around are
cleared). Finally, sys/sysmacros.h has to be included now before
sys/atomic.h, but upstream includes it after so include it here so that
the guards make the out-of-order includes in upstream irrelevant.
Sponsored by: Netflix
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D35750
OpenZFS is based on having a number of layers of include files that
define things and include the next layer. Insert a later at the start
for files in libsa/zfs/spl so we can override issues with upstream
OpenZFS while we work to get the issues corrected upstream.
Sponsored by: Netflix
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D35750
Add a note that veriexec / bearssl builds will pollute the CFLAGS in a
way that's somewhat hard to fix, so I'm just noting it for now.
Sponsored by: Netflix
Only add -DWEAK_REFS to sha256.c and sha512.c instead of
everything. Remove redundant include that's not needed.
Minor formatting tweak.
Sponsored by: Netflix
CFLAGS+= here affects *ALL* libsa files being built. However, this is
only needed for zfs.c, so define it only for this. Also, use the defines
from defs.mk. Move all the zfs.c include hacks together. Also, move the
-Wformat -Wall warnings that were added to CFLAGS+= to the individual
files instead for the same reason.
Sponsored by: Netflix
ZFSSRC is the top level directory where the ZFS sources come from. Don't
reuse it for a list of ZFS sources. Instead, use ZFS_SRC
Sponsored by: Netflix
geli_probe_and_attach() does pick geli_devsw structure for
encrypted disks, the implementation depends on device
name "disk" when device type is DEVT_DISK, but geli_devsw is
setting name field "gelidisk".
PR: 264282
Submitted by: yamagi@yamagi.org
Reported by: yamagi@yamagi.org
MFC after: 2 weeks
Notable upstream pull request merges:
#12321 Fix inflated quiesce time caused by lwb_tx during zil_commit()
#13244 zstd early abort
#13360 Verify BPs as part of spa_load_verify_cb()
#13452 More speculative prefetcher improvements
#13466 Expose zpool guids through kstats
#13476 Refactor Log Size Limit
#13484 FreeBSD: libspl: Add locking around statfs globals
#13498 Cancel in-progress rebuilds when we finish removal
#13499 zed: Take no action on scrub/resilver checksum errors
#13513 Remove wrong assertion in log spacemap
Obtained from: OpenZFS
OpenZFS commit: b9d98453f9
head_errlog is new format for errlog, but we do not really
use errlog, so we can just mark it supported, to enable reading
from pool.
MFC after: 1 week
Move declarations into a new nvlist.h rather than putting everything in
libzfs.h. This makes this nvlist code easier to reuse elsewhere. In
particular, the nvlist implementation in sys/contrib/libnv does not
provide XDR encoding, but this is needed when reading from or writing to
ZFS pools.
Also:
- Remove references to boolean_t. It has to be a 32-bit int here, so
just reference the underlying type.
- Add includes needed when compiling the nvlist code outside of stand/.
No functional change intended.
Reviewed by: tsoome
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35255
When adding an entry to an nvlist, the data buffer might need to be
resized. When this happens, the XDR encoder's notion of the buffer size
also needs to be updated, otherwise the operation may erroneously fail.
Reviewed by: tsoome, imp
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35254
Turns out there is a libsa.3. It's a bit out of date, but we reference
it in a number of places so we should install it. We need to do the DO32
dance because this Makefile is included twice and we don't want it
installing twice.
Sponsored by: Netflix
This mirrors dmu_read_impl(), zeroing out the tail end of the buffer and
clipping the read to what's contained by the block that exists.
This fixes an issue that arose during the 13.1 release process; in
13.1-RC1 and later, setting up GELI+ZFS will result in a failure to
boot. The culprit is this, which causes us to fail to load geom_eli.ko
as there's a residual portion after the single datablk that should be
zeroed out.
PR: 263407
Reviewed by: tsoome
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D35019
Notable upstream pull request merges:
#9078: log xattr=sa create/remove/update to ZIL
#11919: Cross-platform xattr user namespace compatibility
#13014: Report dnodes with faulty bonuslen
#13016: FreeBSD: Fix zvol_cdev_open locking
#13019: spl: Don't check FreeBSD rwlocks for double initialization
#13027: Fix clearing set-uid and set-gid bits on a file when
replying a write
#13031: Add enumerated vdev names to 'zpool iostat -v' and
'zpool list -v'
#13074: Enable encrypted raw sending to pools with greater ashift
#13076: Receive checks should allow unencrypted child datasets
#13098: Avoid dirtying the final TXGs when exporting a pool
#13172: Fix ENOSPC when unlinking multiple files from full pool
Obtained from: OpenZFS
OpenZFS commit: a86e089415
- Use AES_XTS_IV_LEN instead of the key length as the IV length.
- Use G_ELI_IVKEYLEN as the size of the zeroed iv[] array in
g_eli_crypto_cipher() to match geli_io().
PR: 261172
Reported by: Malcolm Matalka <mmatalka@gmail.com>, mikael
Reviewed by: markj
Sponsored by: FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33884
This helps mesuring what's happening when we load the kernel/modules/mfsroot.
This also adds TSENTER2 which uses the third argument of TSRAW, same
as in the kernel.
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33699
Shortlinks occupy the space of both di_db and di_ib when used. However,
everywhere that wants to read or write a shortlink takes a pointer do
di_db and promptly runs off the end of it into di_ib. This is fine on
most architectures, if a little dodgy. However, on CHERI, the compiler
can optionally restrict the bounds on pointers to subobjects to just
that subobject, in order to mitigate intra-object buffer overflows, and
this is enabled in CheriBSD's pure-capability kernels.
Instead, clean this up by inserting a union such that a new di_shortlink
can be added with the right size and element type, avoiding the need to
cast and allowing the use of the DIP macro to access the field. This
also mirrors how the ext2fs code implements extents support, with the
exact same structure other than having a uint32_t i_data[] instead of a
char di_shortlink[].
Reviewed by: mckusick, jhb
Differential Revision: https://reviews.freebsd.org/D33650
tftp_open reads the first block so copy it in the cached data.
If we have more than one block (i.e. we called tftp_read before
tftp_preload) simply just reset the transfer.
Reported by: mmel
Reviewed by: mmel, tsoome
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33697
The preload method will transfer the whole file in a buffer and cache it
so read/lseek operations are faster.
Reviewed by: imp, tsoome
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33410
When we load an ELF file (kernel or module) we do seek(2) a lot to
parse/load the different sections of the ELF file.
Protocol like TFTP suffers a lot from this as there is no resume or
a way to start the tranfer from a specified offset in the file.
fs_preload is added to help those protocol.
Call preload just after opening the ELF file that we need to load so
the underlying method can cache the hole file and then read/lseek operations
are faster.
Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33409
getsecs is very costly, reuse the values we got before.
Fetching a ~30MB kernel with the tftp command use to take ~26 seconds
and now it's ~18 seconds.
Reviewed by: imp, tsoome
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33408
When we load a kernel or module we open/close it a few times.
Since we're using the same port number each time and that we requested
the same file the ACK that we send are valid on the server side and the
server send us the file multiple times.
This makes tftp loading time very inconsistant due to the UDP "flood" that
we have to process.
Reviewed by: imp, tsoome
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33407
readtest will simply load the file in memory, useful for timing
loading on some filesystems.
Reviewed by: tsoome
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33411
tftp-hpa sends NAK with tftp error set to 0 when trying to get
a directory and this is the first thing that loader tries to do
and this make it hangs.
Reviewed by: imp, tsoome
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33406
As part of decommissioning mips support, remove the boot loader
support. Do this in advance of other boot loader work to limit the
amount of work that will be thrown away.
Sponsored by: Netflix
Reviewed by: jrtc27
Differential Revision: https://reviews.freebsd.org/D33377
The STDSB macro is passed to the ffs_sbget() routine to fetch a
UFS/FFS superblock "from the stadard place". It was identically defined
in lib/libufs/libufs.h, stand/libsa/ufs.c, sys/ufs/ffs/ffs_extern.h,
and sys/ufs/ffs/ffs_subr.c. Delete it from these four files and
define it instead in sys/ufs/ffs/fs.h. All existing uses of this macro
already include sys/ufs/ffs/fs.h so no include changes need to be made.
No functional change intended.
Sponsored by: Netflix
net_open() does replace f_devdata with pointer to netdev_sock,
this will cause memory leak when device is closed, but also does
alter the devopen() logic.
We should store &netdev_sock to dev->d_opendata instead, this
would preserve and follow the devopen() logic.
Fixes network boot on aarch64 (tested by bz).
Reviewed-by: imp
MFC After: 2 weeks
Differential Revision: https://reviews.freebsd.org/D32227
Pass the ivlen along through, and just drop this KASSERT() if we're
building _STANDALONE for the time being.
Fixes: 1833d6042c ("crypto: Permit variable-sized IVs ...")
Booting FreeBSD on an EC2 c5.xlarge instance, the loader "twiddles"
810 times over the course of 510 ms, a rate of 1.59 kHz. Even accepting
that many systems are slower than this particular VM and will take
longer to boot (especially if using spinning-rust disks), this seems
like an unhelpfully large amount of twiddling when compared to the
~60 Hz frame rate of many displays; printing the twiddles also consumes
roughly 10% of the boot time on the aforementioned VM.
Setting the default globaldiv to 16 dramatically reduces the time spent
printing twiddles to the console while still twiddling at roughly 100
Hz; this should be ample even for systems which take longer to boot and
consequently twiddle slower.
Note that this can adjusted via the twiddle_divisor variable in
loader.conf, but that file is not processed until nearly halfway
through the loader's runtime.
Reviewed by: allanjude, jrtc27, kevans
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: <https://reviews.freebsd.org/D32163>
Now that the loader tslog code doesn't call printf, we can profile
printf using TSLOG. On an EC2 c5.xlarge instance, we spend roughly
45 ms here (out of roughly 500 ms), presumably due to the time spent
writing output to the console.
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva
We want to keep our root file system open to preserve bcache segment
between file accesses, thus reducing physical disk IO.
Reviewed by: imp, allanjude, kevans (previous version)
Differential Revision: https://reviews.freebsd.org/D30848
MFC after: 1 month
Summary:
Open file list is currently created as statically allocated array (64 items).
Once this array is filled up, loader will not be able to operate with files.
In most cases, this mechanism is good enough, but the problem appears, when
we have many disks with zfs pool(s). In current loader implementation, all
discovered zfs pool configurations are kept in memory and disk devices open -
consuming the open file array. Rewrite the open file mechanism to use
dynamically allocated list.
Reviewed by: imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D31364
servip is set from bootp bp_siaddr (if present) and rootip is
set immediately from servip in tha sane bootp code.
However, the common/dev_net.c does only set rootip (based on
url processing etc). Therefore, we should also use rootip in tftp
reader.
Fixes hung tftp based boot when bp_siaddr is not provided.
MFC after: 1 week
In my initial testing, these are the functions which showed up as
being worth instrumenting. More may be added later.
common/console.c: cons_probe
common/gfx_fb.c: read_list, insert_font, autoload_font
common/interp.c: interact
common/interp_lua.c: interp_init, interp_run
efi/libefi/efipart.c: efipart_readwrite
i386/libi386/biosdisk.c: bd_init, bd_open, bd_edd_io, bd_chs_io, bd_io
libsa/open.c: open
libsa/read.c: read
libsa/twiddle.c: twiddle
Note that profiling interp_run may be of questionable utility as it
may depend on user behaviour (e.g. pressing keys).
Reviewed by: kevans (earlier version)
At present this only supports x86, due to the use of the rdtsc
instruction; and is inert unless a buffer is allocated and passed to
the tslog code (which will be done by a future commit).
Reviewed by: kevans
A bug in the loader's bzipfs & gzipfs filesystems caused compressed
kernel and modules not to work on EFI systems with a veriexec-enabled
loader. Since the size of files in these filesystems are not known
_a priori_ `stat` would initialize the size to -1 and the loader would
then hang in an infinite loop while trying to seek (read) to the end
of file since the loop termination condition compares the current
offset to that negative target position.
Reviewers: vangyzen, imp, Bret Ketchum (Bret.Ketchum@dell.com)
Differential Revision: https://reviews.freebsd.org/D30414
Sponsored by: Dell EMC Isilon
MFC to: stable/12, stable/13
MFC after: 1 week
LLVM12 complains if you change the symbol binding:
`error: _longjmp changed binding to STB_GLOBAL`
In this case LLVM actually ignored the weak directive and used the
later .global, but GNU as would mark the symbol as weak.
None of the other architectures mark the libsa _setjmp as weak so
just drop this directive.
Parentheses added to HASZERO macro to avoid a GCC warning, and formatted
with clang-format as we have adopted these and don't consider them
'contrib' code.
Obtained from: musl (snapshot at commit 4d0a82170a25)
Reviewed by: kib (libc integration), mjg (both earlier)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17630
Notable upstream changes:
778869fa1 Fix reporting of mount progress
e7adccf7f Disable use of hardware crypto offload drivers on FreeBSD
03e02e5b5 Fix checksum errors not being counted on repeated repair
64e0fe14f Restore FreeBSD resource usage accounting
11f2e9a49 Fix panic if scrubbing after removing a slog device
MFC after: 2 weeks
Caller is not interested in symlinks follow them.
Throw an error if too many links encountered.
Reviewed by: stevek
Sponsored by: Juniper Networks
--This line, and those below, will be ignored--
> Description of fields to fill in above: 76 columns --|
> PR: If a GNATS PR is affected by the change.
> Differential Revision: https://reviews.freebsd.org/D### (*full* phabric URL needed).
> Submitted by: If someone else sent in the change.
> Reviewed by: If someone else reviewed your modification.
> Approved by: If you needed approval for this commit.
> Obtained from: If the change is from a third party.
> MFC after: N [day[s]|week[s]|month[s]]. Request a reminder email.
> MFH: Ports tree branch name. Request approval for merge.
> Relnotes: Set to 'yes' for mention in release notes.
> Security: Vulnerability reference (one per line) or description.
> Sponsored by: If the change was sponsored by an organization.
> Empty fields above will be automatically removed.
Change-Id: I4ef92ff256f503c620dc5bba79ed93b32cb2330d
The integer arrays are encoded in nvlist as counted array <count, i0, i1...>,
loader xdr_array() is missing the count. This will affect the pool import when
there are hole devices in pool.
Also fix the new data add and print functions.
Follow-up to r353959 and r368070: do the same for other architectures.
arm32 already seems to use its own .fnstart/.fnend directives, which
appear to be ARM-specific variants of the same thing. Likewise, MIPS
uses .frame directives.
Reviewed by: arichardson
Differential Revision: https://reviews.freebsd.org/D27387
We don't have NEON available in the boot loader, so we have to disable
it. OpenZFS included ZSTD which used the wrong symbol to bring in neon
support. Change to use the code that's been submitted upstream as a
pull request to both.
__ARM_NEON is the proper symbol, defined in ARM C Language Extensions
Release 2.1 (https://developer.arm.com/documentation/ihi0053/d/). Some
sources suggest __ARM_NEON__, but that's the obsolete spelling from
prior versions of the standard.
OpenZFS Pull Request: https://github.com/openzfs/zfs/pull/11055
ZSTD Pull Request: https://github.com/facebook/zstd/pull/2356
We are using asize property from pool label and we do not depend
on partition data to find last two pool labels and to validate LBA for disk IO.
This does allow us to re-enable support for partitionless disk setups.
The OpenZFS code that uses the BMI instructions is broken. Forcibly
disable them to prevent their use. When enabled, the build breaks.
This fixes the build when compiled for a core with BMI instructions.
This is the same fix committed in r364777, for the same issue.
Submitted by: Jung-uk Kim
Add support to the _STANDALONE environment enough bits of the kernel
that we can compile it. We still have a small zstd_shim.c since there
were 3 items that were a bit hard to nail down and may be cleaned up
in the future. These go hand in hand with a number of commits to
sys/sys in the past weeks, should this need be MFCd.
Discussed with: mmacy (in review and on IRC/Slack)
Reviewed by: freqlabs (on openzfs repo)
Differential Revision: https://reviews.freebsd.org/D26218
OpenZFS will start using some of the kernel timekeeping bits
shortly. This implements the bare minimum of that which currently
is just the time_seconds variable.
During devswitch probe, we pick boot pool based on boot disk, if the boot
disk happens to have multiple pools in freebsd-zfs partitions, the current
code does pick last pool from boot disk as boot pool. While there is no
way at that stage to test, the more logical approach would be to pick
first matching pool.
This patch is assuming we do pass pool guid pointer with guid value 0,
this will help us to determine, if the guid value is already set or not.
The general suggestion would be not to share disk between different pools.
Reported by: Alexander Leidinger
There was a small window cp was broken. Work around this by using :>
instead of cp /dev/null. Ideally, we'd keep the cp /dev/null in the
build as a regression test, but doing so breaks people that upgraded
during the cp breakage and this is simpler than bootstrapping a
working cp since there's no good __FreeBSD_version sign posts for
that.
Suggested by: lots of people
Too stubborn for his own good: imp
bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.
By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.
By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.
bootonce dataset name is recorded in boot pool labels, bootenv area.
in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.
bootonce and nextboot features are usable in both UEFI and BIOS boot.
To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).
At this time, only lua loader is updated.
Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512
Move hexdump from stand/common/misc.c to stand/libsa/hexdump.c
(svn cp)
Disable use of pager - causes linking issue for boot1
can be re-enabled by defining HEXDUMP_PAGER.
Reviewed by: stevek, imp
MFC after: 1 week
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D26235
Please note that neither zstd nor encryption is
supported by the loader at this instant. This
change makes it safe to use those features in
one's root pool, but not in one's root dataset.
It can be useful to know what type of machine we are running on for desktop
related thing.
It also allow us to support all the DMI variable that linux driver can fetch.
MFC after: 1 week
Sponsored by: Sponsored-by: The FreeBSD Foundation
The checkpoints are another way of keeping the state of ZFS.
During the rewind, the pool has to be exported.
This makes checkpoints unusable when using ZFS as root.
Add the option to rewind the ZFS checkpoint at the boot time.
If checkpoint exists, a new option for rewinding a checkpoint will appear in
the bootloader menu.
We fully support boot environments.
If the rewind option is selected, the boot loader will show a list of
boot environments that existed before the checkpoint.
Reviewed by: tsoome, allanjude, kevans (ok with high-level overview)
Differential Revision: https://reviews.freebsd.org/D24920
Because we need to read asize from vdev_tree. We also need to consider
different vdev type difference.
Reviewed by: allanjude
Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25586
The DIOCGMEDIASIZE is calculated md->md_sectorsize * md->md_provsize, and
for boot disk, the md_sectorsize is 4k. However, the md_provsize is already
in units of bytes.
Sponsored by: Netflix, Klara Inc.
The vdev_init() does check for "known" vdev types, the [log] device removal will create "hole"
device, but vdev_init() does not allow it.
Obtained from: illumos
MFC after: 1 week
We should have nextboot feature implemented in libsa zfs code.
To get there, I have created zfs_nextboot() implementation based on
two sources, our current simple textual string based approach with added
structured boot label PAD structure from OpenZFS.
Secondly, all nvlist details are moved to separate source file and
restructured a bit. This is done to provide base support to add nvlist
add/update feature in followup updates.
And finally, the zfsboot/gptzfsboot disk access functions are swapped to use
libi386 and libsa.
Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25324
fs_summary_info structure. This change was originally done
by the CheriBSD project as they need larger pointers that
do not fit in the existing superblock.
This cleanup of the superblock eases the task of the commit
that immediately follows this one.
Suggested by: brooks
Reviewed by: kib
PR: 246983
Sponsored by: Netflix
- Use enc_xform_aes_xts.setkey() directly instead of duplicating the code
now that it no longer calls malloc().
- Rather than bringing back all of xform_userland.h, add a conditional
#include of <stand.h> to xform_enc.h.
- Update calls to encrypt/decrypt callbacks in enc_xform_aes_xts for
separate input/output pointers.
Pointy hat to: jhb
Assume gcc is at least 6.4, the oldest xtoolchain in the ports tree.
Assume clang is at least 6, which was in 11.2-RELEASE. Drop conditions
for older compilers.
Reviewed by: imp (earlier version), emaste, jhb
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D24802
When reading less than sector size but from sector boundary,
the vdev_read() will read full sector into the provided buffer
and therefore corrupting memory past buffer end.
MFC after: 2 days
This code was trying to use a pointer value for st_dev, which is definitely
not a pointer. Instead, cast to uintptr_t so it becomes a non-pointer value
before casting it.
Tested: mips-gcc cross compile, mips32 build
On slow platforms, it helps to spread the hashing load
over time so that tftp does not timeout.
Also, some .4th files are too big to fit in cache of pkgfs,
so increase cache size and ensure fully populated.
Reviewed by: stevek
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D24287
This option is intended to aid development, to allow building with user malloc.
The use case would be to build userboot & test with libc (or other) malloc and
use extra malloc debug features.
We need a valid st_dev, st_ino and st_mtime
to correctly track which files have been verified
and to update our notion of time.
ve_utc_set(): ignore utc if it would jump our current time
by more than VE_UTC_MAX_JUMP (20 years).
Allow testing of install command via userboot.
Need to fix its stat implementation too.
bhyveload also needs stat fixed - due to change to userboot.h
Call ve_error_get() from vectx_close() when hash is wrong.
Track the names of files we have hashed into pcr
For the purposes of measured boot, it is important
to be able to reproduce the hash reflected in
loader.ve.pcr
so loader.ve.hashed provides a list of names in the order they
were added.
Reviewed by: imp
MFC after: 1 week
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org//D24027
we have crc32(const void *, size_t) in libsa. Unfortunately zlib has
crc32(long, const unigned char *, unsigned) and we have conflict.
Since we do build libsa with zlib, we can use zlib version instead.
Reviewed by: allanjude
Differential Revision: https://reviews.freebsd.org/D24068
smbios used to be an i386 only kinda weird quirk to the x86
architecture. But UEFI picked it up, dusted it off and now it's many
other locations. Make it base technology by moving it to libsa and
fixing up the compliation. The code has issues with unaligned access
still, but that will be addressed in a followup commit.
Differential Revision: https://reviews.freebsd.org/D23660
Create a ptov() function. It's basically the same as the btx PTOV
macro, but works everywhere. smbios needs this to translate addresses,
but the translation differs between BIOS booting and EFI booting. Make
it a function so one smbios.o can be used everywhere. Provide
definitions for it in the two loaders affected.
Differential Revision: https://reviews.freebsd.org/D23660
This uses the new layout of the upstream repository, which was recently
migrated to GitHub, and converted into a "monorepo". That is, most of
the earlier separate sub-projects with their own branches and tags were
consolidated into one top-level directory, and are now branched and
tagged together.
Updating the vendor area to match this layout is next.
In some cases the pool discovery will get stuck in infinite loop while setting
up the vdev children.
To fix, we split the vdev setup into two parts, first we create vdevs based on
configuration we do get from pool label, then, we process pool config from MOS
and update the pool config if needed.
Testing done: confirm previously hung loader is not hung any more.
MFC after: 1 week
Instead of passing NULL to fzap_name_equal and crashing, just return
ENOENT. This happened when higher bits of a hash of the searched key
(its hash prefix) matched a hash prefix of some key in the ZAP, but the
full hash value of the searched key did not match any key in the ZAP.
I observerved this problem when loader tried to look up
"features_for_read" in a particular old pool that predates pool
features.
MFC after: 2 weeks
Sponsored by: Panzura
We do allocate amount of memory (void * or char *), and then assign this
buffer to struct iso_primary_descriptor *vd. Make sure we do
allocate enough bytes.
In fact we do allocate enough, but it is good idea to make sure this really
is so.
MFC after: 1 week
o Remove All Rights Reserved from my notices
o imp@FreeBSD.org everywhere
o regularize punctiation, eliminate date ranges
o Make sure that it's clear that I don't claim All Rights reserved by listing
All Rights Reserved on same line as other copyright holders (but not
me). Other such holders are also listed last where it's clear.
illumos/illumos-gate@663207adb1663207adb1
10601 Pool allocation classes
https://www.illumos.org/issues/10601
illumos port of ZoL Pool allocation classes. Includes at least these two
commits:
441709695 Pool allocation classes misplacing small file blocks
cc99f275a Pool allocation classes
10757 Add -gLp to zpool subcommands for alt vdev names
https://www.illumos.org/issues/10757
Port from ZoL of
d2f3e292d Add -gLp to zpool subcommands for alt vdev names
Note that a subsequent ZoL commit changed -p to -P
a77f29f93 Change full path subcommand flag from -p to -P
Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Portions contributed by: Håkan Johansson <f96hajo@chalmers.se>
Portions contributed by: Richard Yao <ryao@gentoo.org>
Portions contributed by: Chunwei Chen <david.chen@nutanix.com>
Portions contributed by: loli10K <ezomori.nozomu@gmail.com>
Author: Don Brady <don.brady@delphix.com>
11541 allocation_classes feature must be enabled to add log device
illumos/illumos-gate@c1064fd7cec1064fd7cehttps://www.illumos.org/issues/11541
After the allocation_classes feature was integrated, one can no longer add a
log device to a pool unless that feature is enabled. There is an explicit check
for this, but it is unnecessary in the case of log devices, so we should handle
this better instead of forcing the feature to be enabled.
Author: Jerry Jelinek <jerry.jelinek@joyent.com>
FreeBSD notes.
I faithfully added the new -g, -L, -P flags, but only -g does something:
vdev GUIDs are displayed instead of device names. -L, resolve symlinks,
and -P, display full disk paths, do nothing at the moment.
The use of special vdevs is backward compatible for read-only access, so
root pools should be bootable, but exercise caution.
MFC after: 4 weeks
Include the server IP address when logging nfs_open(), add a few missing
"\n"s, and correct a typo.
Reviewed by: kevans
MFC after: 2 weeks
Sponsored by: Panasas
Differential Revision: https://reviews.freebsd.org/D22346
In our case the structure is more complex and simple static initializer
will upset compiler diagnostics - using memset is still better than building
more complext initializer.
Since physical device asize is calculated from psize and the asize is stored
in pool label, we can use asize to set the value of psize, which is used to
calculate the location of the pool labels.
MFC after: 1 week
Port illumos change: https://www.illumos.org/issues/11667
Move lz4.c out of zfs tree to opensolaris/common/lz4, adjust it to be
usable from kernel/stand/userland builds, so we can use just one single
source. Add lz4.h to declare lz4_compress() and lz4_decompress().
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D22037
Add ficl words for isvirtualized
and move ficl inb and outb words to ficl/x86/sysdep.c
so can be shared by i386 and amd64
Reviewed by: imp bdrewery
MFC after: 1 week
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D22069
