When a temporary/bootonce boot environment is renamed, we need to also
update the bootenv nvlist on-disk to reflect the new name. Additionally,
when a temporary/bootonce boot environment is destroyed, we also need to
clear out the on-disk state.
Reviewed by: kevans
Approved by: kp
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D43591
Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.
Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/
Sponsored by: Netflix
beadm will recursively promote deep BE datasets. In order to match the
beadm behavior, we need to recursively iterate over child filesystems
and promote them along the way.
This patch further refines the work from D40903, completing the fix for
promotion.
Reviewed by: kevans, rew
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D40972
This matches the beadm behavior; generally, we need to keep promoting
until the BE is no longer a clone from a snapshot. This fixes scenarios
where the dataset associated with a BE's origin is itself a clone,
activating the BE previously would promote it to a clone of the origin's
origin.
We could keep using be_get_dataset_props here, except for two
annoyances:
1.) I couldn't find a clean way to just clear an nvlist rather than
having to re-alloc it, and I didn't want to just remove the one prop
we're inspecting out of it.
2.) That's a lot of overhead when all we want to do is fetch the origin
anyways.
Note that this is not a complete fix, but it does fix the majority of
cases; deep BE subordinates are still notably broken, pending a patch
from Christian.
Reported by: R. Christian McDonald <rcm@rcm.sh>
Reviewed by: rew
Differential Revision: https://reviews.freebsd.org/D40903
As noted in the comment, we already know the rest of libbe_init() will
fail because there's no pool imported. Avoid the side effect by
checking beforehand and bailing out early.
With this, freebsd-update(8) should no longer trigger a load of the zfs
kmod just because it runs `bectl check`.
Reviewed by: jwmaag_gmail.com, rew
Differential Revision: https://reviews.freebsd.org/D36188
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
Notable upstream pull request merges:
#11680 Add support for zpool user properties
#14145 Storage device expansion "silently" fails on degraded vdev
#14405 Create zap for root vdev
#14659 Allow MMP to bypass waiting for other threads
#14674 Miscellaneous FreBSD compilation bugfixes
#14692 Fix some signedness issues in arc_evict()
#14702 Fix typo in check_clones()
#14715 module: small fixes for FreeBSD/aarch64
#14716 Trim needless zeroes from checksum events
#14719 vdev: expose zfs_vdev_max_ms_shift as a module parameter
#14722 Fix "Detach spare vdev in case if resilvering does not happen"
#14723 freebsd clone range fixes
#14728 Fix BLAKE3 aarch64 assembly for FreeBSD and macOS
#14735 Fix in check_filesystem()
#14739 Fix data corruption when cloning embedded blocks
#14758 Fix VERIFY(!zil_replaying(zilog, tx)) panic
#14761 Revert "ZFS_IOC_COUNT_FILLED does unnecessary txg_wait_synced()"
#14774 FreeBSD .zfs fixups
#14776 FreeBSD: make zfs_vfs_held() definition consistent with declaration
#14779 powerpc64: Support ELFv2 asm on Big Endian
#14788 FreeBSD: add missing vop_fplookup assignments
#14789 PAM: support the authentication facility
#14790 Revert "Fix data race between zil_commit() and zil_suspend()"
#14795 Fix positive ABD size assertion in abd_verify()
#14798 Mark TX_COMMIT transaction with TXG_NOTHROTTLE
#14804 Correct ABD size for split block ZIOs
#14806 Use correct block pointer in block cloning case.
#14808 blake3: fix up bogus checksums in face of cpu migration
Obtained from: OpenZFS
OpenZFS commit: d96e29576c
Notable upstream pull request merges:
#12194 Fix short-lived txg caused by autotrim
#13368 ZFS_IOC_COUNT_FILLED does unnecessary txg_wait_synced()
#13392 Implementation of block cloning for ZFS
#13741 SHA2 reworking and API for iterating over multiple implementations
#14282 Sync thread should avoid holding the spa config write lock
when possible
#14283 txg_sync should handle write errors in ZIL
#14359 More adaptive ARC eviction
#14469 Fix NULL pointer dereference in zio_ready()
#14479 zfs redact fails when dnodesize=auto
#14496 improve error message of zfs redact
#14500 Skip memory allocation when compressing holes
#14501 FreeBSD: don't verify recycled vnode for zfs control directory
#14502 partially revert PR 14304 (eee9362a7)
#14509 Fix per-jail zfs.mount_snapshot setting
#14514 Fix data race between zil_commit() and zil_suspend()
#14516 System-wide speculative prefetch limit
#14517 Use rw_tryupgrade() in dmu_bonus_hold_by_dnode()
#14519 Do not hold spa_config in ZIL while blocked on IO
#14523 Move dmu_buf_rele() after dsl_dataset_sync_done()
#14524 Ignore too large stack in case of dsl_deadlist_merge
#14526 Use .section .rodata instead of .rodata on FreeBSD
#14528 ICP: AES-GCM: Refactor gcm_clear_ctx()
#14529 ICP: AES-GCM: Unify gcm_init_ctx() and gmac_init_ctx()
#14532 Handle unexpected errors in zil_lwb_commit() without ASSERT()
#14544 icp: Prevent compilers from optimizing away memset()
in gcm_clear_ctx()
#14546 Revert zfeature_active() to static
#14556 Remove bad kmem_free() oversight from previous zfsdev_state_list
patch
#14563 Optimize the is_l2cacheable functions
#14565 FreeBSD: zfs_znode_alloc: lock the vnode earlier
#14566 FreeBSD: fix false assert in cache_vop_rmdir when replaying ZIL
#14567 spl: Add cmn_err_once() to log a message only on the first call
#14568 Fix incremental receive silently failing for recursive sends
#14569 Restore ASMABI and other Unify work
#14576 Fix detection of IBM Power8 machines (ISA 2.07)
#14577 Better handling for future crypto parameters
#14600 zcommon: Refactor FPU state handling in fletcher4
#14603 Fix prefetching of indirect blocks while destroying
#14633 Fixes in persistent error log
#14639 FreeBSD: Remove extra arc_reduce_target_size() call
#14641 Additional limits on hole reporting
#14649 Drop lying to the compiler in the fletcher4 code
#14652 panic loop when removing slog device
#14653 Update vdev state for spare vdev
#14655 Fix cloning into already dirty dbufs
#14678 Revert "Do not hold spa_config in ZIL while blocked on IO"
Obtained from: OpenZFS
OpenZFS commit: 431083f75b
be_clone calls be_clone_cb and both call set_error on the return
error path. set_error prints the error resulting in a double print.
be_clone_cb should just return the error code and allow be_clone
to print it.
PR: 265248
Reported by: Graham Perrin
Reviewed by: imp, kevans
Pull Request: https://github.com/freebsd/freebsd-src/pull/685
Notable upstream pull request merges:
#13805 Configure zed's diagnosis engine with vdev properties
#14110 zfs list: Allow more fields in ZFS_ITER_SIMPLE mode
#14121 Batch enqueue/dequeue for bqueue
#14123 arc_read()/arc_access() refactoring and cleanup
#14159 Bypass metaslab throttle for removal allocations
#14243 Implement uncached prefetch
#14251 Cache dbuf_hash() calculation
#14253 Allow reciever to override encryption property in case of replication
#14254 Restrict visibility of per-dataset kstats inside FreeBSD jails
#14255 Zero end of embedded block buffer in dump_write_embedded()
#14263 Cleanups identified by CodeQL and Coverity
#14264 Miscellaneous fixes
#14272 Change ZEVENT_POOL_GUID to ZEVENT_POOL to display pool names
#14287 FreeBSD: Remove stray debug printf
#14288 Colorize zfs diff output
#14289 deadlock between spa_errlog_lock and dp_config_rwlock
#14291 FreeBSD: Fix potential boot panic with bad label
#14292 Add tunable to allow changing micro ZAP's max size
#14293 Turn default_bs and default_ibs into ZFS_MODULE_PARAMs
#14295 zed: add hotplug support for spare vdevs
#14304 Activate filesystem features only in syncing context
#14311 zpool: do guid-based comparison in is_vdev_cb()
#14317 Pack zrlock_t by 8 bytes
#14320 Update arc_summary and arcstat outputs
#14328 FreeBSD: catch up to 1400077
#14376 Use setproctitle to report progress of zfs send
#14340 Remove some dead ARC code
#14358 Wait for txg sync if the last DRR_FREEOBJECTS might result in a hole
#14360 libzpool: fix ddi_strtoull to update nptr
#14364 Fix unprotected zfs_znode_dmu_fini
#14379 zfs_receive_one: Check for the more likely error first
#14380 Cleanup of dead code suggested by Clang Static Analyzer
#14397 Avoid passing an uninitialized index to dsl_prop_known_index
#14404 Fix reading uninitialized variable in receive_read
#14407 free_blocks(): Fix reports from 2016 PVS Studio FreeBSD report
#14418 Introduce minimal ZIL block commit delay
#14422 x86 assembly: fix .size placement and replace .align with .balign
Obtained from: OpenZFS
OpenZFS commit: 9cd71c8604
It is useful to have zfs utilities and lib in a separate package as
it allow users to create image that can support ZFS (i.e. not with
WITHOUT_ZFS in src.conf set) without bloating the default image with
all zfs tools (for example for jails).
Differential Revision: https://reviews.freebsd.org/D36225
This check was previously in `create` only, not applying to renames. It
should really be applied at the libbe level, so that we can avoid
writing about this restriction over and over again.
While we're here: `bectl rename` always succeeds, even when it doesn't.
Start returning the error.
Reported By: Christian McDonald <cmcdonald netgate com>
Reviewed by: rew, jwmaag_gmail.com (earlier version)
Differential Revision: https://reviews.freebsd.org/D34605
The props feature to this API was added as a "just in case", with no
planned consumer. It turns out that it was implemented incorrectly,
tapping out props for the BE root (BE's parent) rather than the BE
itself in all cases.
Fetch properties for the dataset that's actually mounted at the queried
path.
Reported by: Christian McDonald <cmcdonald netgate com>
MFC after: 3 days
Notable upstream pull request merges:
#12766 Fix error propagation from lzc_send_redacted
#12805 Updated the lz4 decompressor
#12851 FreeBSD: Provide correct file generation number
#12857 Verify dRAID empty sectors
#12874 FreeBSD: Update argument types for VOP_READDIR
#12896 Reduce number of arc_prune threads
#12934 FreeBSD: Fix zvol_*_open() locking
#12947 lz4: Cherrypick fix for CVE-2021-3520
#12961 FreeBSD: Fix leaked strings in libspl mnttab
#12964 Fix handling of errors from dmu_write_uio_dbuf() on FreeBSD
#12981 Introduce a flag to skip comparing the local mac when raw sending
#12985 Avoid memory allocations in the ARC eviction thread
Obtained from: OpenZFS
OpenZFS commit: 17b2ae0b24
the current one first. And if it fails to do so, it abandons activation.
However, with the new bootonce feature, there is a legitimate case when
a pool doesn't have "bootfs" property set. Check for this case before
calling be_deactivate().
Reviewed by: kevans
runtime contain what is needed to boot in single user and repair a
system, bectl could be handy to have in this situation.
Differential Revision: https://reviews.freebsd.org/D27708
While toying around with lua bindings for libbe(3), I discovered that I
apparently never documented this, despite having documented
be_is_auto_snapshot_name that references it.
MFC after: 1 week
libbe will never need to mutate these as we either process them into a local
buffer or we just don't touch them and write to a separate out argument.
MFC after: 1 week
bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.
By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.
By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.
bootonce dataset name is recorded in boot pool labels, bootenv area.
in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.
bootonce and nextboot features are usable in both UEFI and BIOS boot.
To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).
At this time, only lua loader is updated.
Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512
sys/ccompile.h no longer uses #pragma ident, so we no longer need to worry
about unknown pragmas.
I fixed one WARNS issue in r363409 by annotating be_is_auto_snapshot_name's
lbh parameter __unused, then upstreamed the following changes to OpenZFS
that rode in with the merge:
- zfs_path_to_zhandle now takes a const char *path rather than a char *path,
since it won't be mutating the string it receives and I had no reason to
believe it will need to in the future. [OpenZFS PR #10605]
- Annotated some unused parameters on definitions inlined into headers as
such. [OpenZFS PR #10606]
The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.
I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.
Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.
Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.
Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872
lbh is included for consistency with other functions and in case
future work needs to use it, but it is currently unused. Mark it,
and a post-OpenZFS-import world will be able to raise WARNS of
libbe to the default (pending some minor changes to openzfs libzfs).
MFC after: 3 days
I sent out an e-mail on 2020/01/21 with a plan to do this to Kyle, Rob, and
Wes; all parties have responded in the affirmative that it's OK to drop it
from these files.
More background is available in r356876, but this new interface is more
portable across ZFS implementations and cleaner for what libbe is attempting
to achieve anyways.
MFC after: 3 days
When removing a boot environment iterate over the dependents and process the
snapshots by grabbing any clones. Promote the clones we found and then
remove the target environment.
This fixes the ability to destroy a boot environment when it has been used
to spawn one or more other boot environments.
PR: 242592
Submitted by: Wes Maag <jwmaag gmail com> (with changes by myself)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D22953
Update a bunch of Makefile.depend files as
a result of adding Makefile.depend.options files
Reviewed by: bdrewery
MFC after: 1 week
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D22494
This is the half of the changes required that work as-is with both in-tree
ZFS and the new hotness, sysutils/openzfs. Highlights are less dependency
on header pollution (from somewhere) and using 'mnttab' instead of
'extmnttab'. In the in-tree ZFS, the latter is a #define for the former,
but in the port extmnttab is actually a distinct struct that's a super-set
of mnttab. We really want mnttab here anyways, so just use it.
Imported BE, much like the activated BE, will not have an origin that we can
fetch/examine for destruction. be_destroy should not return BE_ERR_NOORIGIN
for failure to get the origin property for BE_DESTROY_AUTOORIGIN, because
we don't really know going into it that there's even an origin to be
destroyed.
BE_DESTROY_NEEDORIGIN has been renamed to BE_DESTROY_WANTORIGIN because only
a subset of it *needs* the origin, so 'need' is too strong of verbiage.
This was caught by jenkins and the bectl tests, but kevans failed to run the
bectl tests prior to commit.
Reported by: lwhsu
New BEs can be created from either an existing snapshot or an existing BE.
If an existing BE is chosen (either implicitly via 'bectl create' or
explicitly via 'bectl create -e foo bar', for instance), then bectl will
create a snapshot of the current BE or "foo" with be_snapshot, with a name
formatted like: strftime("%F-%T") and a serial added to it.
This commit adds the needed bits for libbe or consumers to determine if a
snapshot names matches one of these auto-created snapshots (with some light
validation of the date/time/serial), and also a be_destroy flag to specify
that the origin should be automatically destroyed if possible.
A future commit to bectl will specify BE_DESTROY_AUTOORIGIN by default so we
clean up the origin in the most common case, non-user-managed snapshots.
The default package use to be FreeBSD-runtime but it should only contain
binaries and libs enough to boot to single user and repair the system, it
is also very handy to have a package that can be tranform to a small mfsroot.
So create a new package named FreeBSD-utilities and make it the default one.
Also move a few binaries and lib into this package when it make sense.
Reviewed by: bapt, gjb
Differential Revision: https://reviews.freebsd.org/D21506
Further cleanup after r349380; loader and kernel will both ignore canmount
on the root dataset as well, so we should not be so strict about it when
mounting it. be_mount is restructured to make it more clear that depth==0 is
special, and to not try fetching these properties that we won't care about.
MFC after: 3 days
Other parts of libbe(3) were fairly strict on the mountpoint property of the
BE dataset, and be_mount was not much better. It was improved in r347027 to
allow mountpoint=none for depth==0, but this bit was still sensitive to
mountpoint != / and mountpoint != none. Given that other parts of libbe(3)
no longer restrict the mountpoint property here, and the rest of the base
system is generally OK and will assume that a BE is mounted at /, let's do
the same.
Reported by: ler
MFC after: 3 days
