If ZFS reports that a disk had at least 8 I/O operations over 60s that
were each delayed by at least 30s (implying a queue depth > 4 or I/O
aggregation, obviously), fault that disk. Disks that respond this
slowly can degrade the entire system's performance.
MFC after: 2 weeks
Sponsored by: Axcient
Reviewed by: delphij
Differential Revision: https://reviews.freebsd.org/D42825
Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.
Sponsored by: Netflix
When run in a jail, /dev/mdctl is missing. So skip any tests that use
mdconfig or mdmfs with md in this case: they can't possibly work. This
is in line with other tests that test for presence of required features
and skip if they aren't present. I did this instead of checking for
jails so they can still run in jails that allow creation of md devices.
Sponsored by: Netflix
* Wait for gnop devices to disappear after "gnop destroy".
Apparently that process is asynchronous now, or maybe it's just slower
than it used to be. Also, after removing a gnop wait for its pool to
be degraded. That isn't instant.
* The zfsd tests no longer require camcontrol.
This was a harmless oversight from
11ed0a95bf
* Fix the zfsd_degrade_001_pos test for recent zfs versions.
ZFS now rate limits checksum errors to about 20 per second. But
zfsd's threshold for degrading a disk is 50 per minute. So we must
alternately corrupt and scrub the pool to ensure that checksum errors
are generated in multiple 1-second windows, so that zfsd will see
enough of them.
* Fix the zfsd_fault_001_pos test in VMs
And, for that matter, when using NVME or SATA disks. As originally
written, the test used the da driver to inject errors. Rewrite it to
use gnop vdevs. gnop can also inject errors. It works on top of any
disk device, and it's also faster than using da.
MFC after: 2 weeks
Sponsored by: Axcient
Differential Revision: https://reviews.freebsd.org/D39437
In most cases, usage does not return, so mark them as __dead2. For the
cases where they do return, they have not been marked __dead2.
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/735
It only ever worked by accident.
* Actually set autoreplace=off as stated in the description
* Wait for the removed device to disappear from the pool before
proceeding.
* In the assertion, verify that the new disk does not get added, as
described in the description.
MFC after: 2 weeks
Sponsored by: Axcient
If you remove an unused spare and then reinsert it, zfsd will now online
it in all pools.
Do not MFC without 2a58b312b6 (but it's ok to MFC that one without this
one).
Submitted by: Ameer Hamza <ahamza@ixsystems.com> (zfsd), Me (tests)
MFC after: 2 weeks
MFC with: 2a58b312b6
Sponsored by: iX Systems, Axcient
Pull Request: https://github.com/freebsd/freebsd-src/pull/697
TL;DR: Three ZFS tests created ZFS pools on all unmounted devices listed
in /etc/fstab, corrupting their contents. Stop that.
Imagine my surprise when the ESP on my main dev/test VM would "randomly"
become corrupted, making it unbootable. Three tests collect various devices
from the system and try to add them to a test pool. The test expects this
to fail because it _assumes_ these devices are in use and ZFS will correctly
reject the request.
My /etc/fstab has two entries for devices in /dev:
/dev/gpt/swap0 none swap sw,trimonce,late
/dev/gpt/esp0 /boot/efi msdosfs rw,noauto
Note the `noauto` on the ESP. In a remarkable example of irony, I chose
this because it should keep the ESP more protected from corruption;
in fact, mounting it would have protected it from this case.
The tests added all of these devices to a test pool in a _single command_,
expecting the command to fail. The swap device was in use, so the command
correctly failed, but the ESP was added and therefore corrupted. However,
since the command correctly failed, the test didn't notice the ESP problem.
If each device had been added with its own command, the test _might_ have
noticed that one of them incorrectly succeeded. However, two of these
tests would not have noticed:
hotspare_create_001_neg was incorrectly specified as needing the Solaris
dumpadm command, so it was skipped. _Some_ of the test needs that command,
but it checks for its presence and runs fine without it.
Due to bug 241070, zpool_add_005_pos was marked as an expected failure.
Due to the coarse level of integration with ATF, this test would still
"pass" even if it failed for the wrong reason. I wrote bug 267554 to
reconsider the use of atf_expect_fail in these tests.
Let's further consider the use of various devices found around the system.
In addition to devices in /etc/fstab, the tests also used mounted devices
listed by the `mount` command. If ZFS behaves correctly, it will refuse
to added mounted devices and swap devices to a pool. However, these are
unit tests used by developers to ensure that ZFS still works after they
modify it, so it's reasonable to expect ZFS to do the _wrong_ thing
sometimes. Using random host devices is unsafe.
Fix the root problem by using only the disks provided via the "disks"
variable in kyua.conf. Use one to create a UFS file system and mount it.
Use another as a swap device. Use a third as a dump device, but expect
it to fail due to bug 241070.
While I'm here:
Due to commit 6b6e2954dd, we can simply add a second dump device and
remove it in cleanup. We no longer need to save, replace, and restore the
pre-existing dump device.
The cleanup_devices function used `camcontrol inquiry` to distinguish disks
from other devices, such as partitions. That works fine for SCSI, but not
for ATA or VirtIO block. Use `geom disk list` instead.
PR: 241070
PR: 267554
Reviewed by: asomers
Sponsored by: Dell Inc.
Differential Revision: https://reviews.freebsd.org/D37257
Merge commit 1737d8397a0 by Brooks Davis:
time() is declared in time.h
This fixes a -Werror warning from clang 15:
tests/sys/cddl/zfs/bin/readmmap.c:97:9: error: call to undeclared function 'time'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
Obtained from: https://github.com/CTSRD-CHERI/cheribsd/commit/1737d8397a0
MFC after: 3 days
And drop stray 'd' from the end of some printed numbers. I assume this
was the result of someone thinking u is a printf length modifier for d,
not a format specifier itself.
Reviewed by: kevans, rew
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34387
The argument has to be a single whitespace-separate value. While touching
all these lines also add ksh93, since `atf_set "require.progs"` overrides
the default value specified in the Kyuafile. This then results in tests
being executed despite ksh93 not being installed.
Reviewed By: asomers
Differential Revision: https://reviews.freebsd.org/D29066
The current default is provided in various Makefile.inc in some top-level
directories and covers a good portion of the tree, but doesn't cover parts
of the build a little deeper (e.g. libcasper).
Provide a default in src.sys.mk and set WARNS to it in bsd.sys.mk if that
variable is defined. This lets us relatively cleanly provide a default WARNS
no matter where you're building in the src tree without breaking things
outside of the tree.
Crunchgen has been updated as a bootstrap tool to work on this change
because it needs r365605 at a minimum to succeed. The cleanup necessary to
successfully walk over this change on WITHOUT_CLEAN builds has been added.
There is a supplemental project to this to list all of the warnings that are
encountered when the environment has WARNS=6 NO_WERROR=yes:
https://warns.kevans.dev -- this project will hopefully eventually go away
in favor of CI doing a much better job than it.
Reviewed by: emaste, brooks, ngie (all earlier version)
Reviewed by: emaste, arichardson (depend-cleanup.sh change)
Differential Revision: https://reviews.freebsd.org/D26455
Messing with gnop devices under a zpool fails in this test, causing
the pool to be suspended and eventually the system to deadlock.
Skip the test for now until the issue is resolved.
PR: tests/248910
Discussed with: lwhsu
Sponsored by: iXsystems, Inc.
The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.
I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.
Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.
Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.
Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872
* Don't partition a disk if too few are available. Just rely on Kyua to
ensure that the tests aren't run with insufficient disks.
* Remove redundant cleanup steps
* In zpool_add_003_pos, store the temporary file in $PWD so Kyua will
automatically clean it up.
* Update zpool_add_005_pos to use dumpon instead of dumpadm. This test had
never been ported to FreeBSD.
* In zpool_add_005_pos, don't format the dump disk with UFS. That was
pointless.
MFC after: 2 weeks
Sponsored by: Axcient
The test is necessarily racy, because it depends on being able to complete a
"zpool add" before a previous resilver finishes. But it was racier than it
needed to be. Move the first "zpool add" to before the resilver starts.
MFC after: 2 weeks
Sponsored by: Axcient
The ZFS test suite was overriding the common $PWD variable with the path to
the pwd command, even though no test wanted to use it that way. Most tests
didn't notice, because ksh93 eventually restored it to its proper meaning.
MFC after: 2 weeks
Sponsored by: Axcient
* Don't create a UFS mountpoint just to store some temporary files. The
tests should always be executed with a sufficiently large TMPDIR.
Creating the UFS mountpoint is not only unneccessary, but it slowed
zpool_import_missing_002_pos greatly, because that test moves large files
between TMPDIR and the UFS mountpoint. This change also allows many of
the tests to be executed with just a single test disk, instead of two.
* Move zpool_import_missing_002_pos's backup device dir from / to $PWD to
prevent cross-device moves. On my system, these two changes improved that
test's speed by 39x. It should also prevent ENOSPC errors seen in CI.
* If insufficient disks are available, don't try to partition one of them.
Just rely on Kyua to skip the test. Users who care will configure Kyua
with sufficient disks.
MFC after: 2 weeks
Sponsored by: Axcient
It was trying to destroy the pool while zfsd was detaching the spare, and
"zpool destroy" failed. Fix by waiting until the spare has fully detached.
MFC after: 2 weeks
Sponsored by: Axcient
The test declared that it only needed 5 disks, but actually tried to use 6.
Fix it to use just 5, which is all it really needs.
MFC after: 2 weeks
Sponsored by: Axcient
* Fix force_sync_path, which ensures that a file is fully flushed to disk.
Apparently "zpool history"'s performance has improved, but exporting and
importing the pool still works.
* Fix file_dva by using undocumented zdb syntax to clarify that we're
interested in the pool's root file system, not the pool itself. This
should also fix the zpool_clear_001_pos test.
* Remove a redundant cleanup step
MFC after: 2 weeks
Sponsored by: Axcient
Differential Revision: https://reviews.freebsd.org/D21901
These tests have never worked correctly
* Replace runwattr with sudo
* Fix a scoping bug with the "dtst" variable
* Cleanup user properties created during tests
* Eliminate the checks for refreservation and send support. They will always
be supported.
* Fix verify_fs_snapshot. It seemed to assume that permissions would not yet
be delegated, but that's not how it's actually used.
* Combine verify_fs_promote with verify_vol_promote
* Remove some useless sleeps
* Fix backwards condition in verify_vol_volsize
* Remove some redundant cleanup steps in the tests. cleanup.ksh will handle
everything.
* Disable some parts of the tests that FreeBSD doesn't support:
* Creating snapshots with mkdir
* devices
* shareisci
* sharenfs
* xattr
* zoned
The sharenfs parts could probably be reenabled with more work to remove the
Solarisms.
MFC after: 2 weeks
Sponsored by: Axcient
Differential Revision: https://reviews.freebsd.org/D21898
ZFS has grown some additional properties that hadn't been added to the
config file yet. While I'm here, improve the error message, and remove a
superfluous command.
MFC after: 2 weeks
Sponsored by: Axcient
This test attempts to corrupt a file-backed vdev by deleting it and then
recreating it with truncate. But that doesn't work, because the pool
already has the vdev open, and it happily hangs on to the open-but-deleted
file. Fix by truncating the file without deleting it.
MFC after: 2 weeks
Sponsored by: Axcient
* Adapt zvol_misc_001_neg to use dumpon instead of Solaris's dumpadm
* Disable zvol_misc_003_neg, zvol_misc_005_neg, and zvol_misc_006_pos,
because they involve using a zvol as a dump device, which FreeBSD does not
yet support.
MFC after: 2 weeks
Sponsored by: Axcient
* Remove zpool_create_013_neg. FreeBSD doesn't have an equivalent of
Solaris's metadevices. GEOM would be the equivalent, but since all geoms
are the same from ZFS's perspective, this test would be redundant with
zpool_create_012_neg
* Remove zpool_create_014_neg. FreeBSD does not support swapping to regular
files.
* Remove zpool_create_016_pos. This test is redundant with literally every
other test that creates a disk-backed pool.
* s:/etc/vfstab:/etc/fstab in zpool_create_011_neg
* Delete the VTOC-related portion of zpool_create_008_pos. FreeBSD doesn't
use VTOC.
* Replace dumpadm with dumpon and swap with swapon in multiple tests.
* In zpool_create_015_neg, don't require "zpool create -n" to fail. It's
reasonable for that variant to succeed, because it doesn't actually open
the zvol.
* Greatly simplify zpool_create_012_neg. Make it safer, too, but not
interfering with the system's regular swap devices.
* Expect zpool_create_011_neg to fail (PR 241070)
* Delete some redundant cleanup steps in various tests
* Remove some unneeeded ATF timeout specifications. The default is fine.
PR: 241070
MFC after: 2 weeks
Sponsored by: Axcient
This mutes the duplicate target warning emitted via bsd.files.mk each build.
MFC after: 1 week
Reviewed by: asomers
Approved by: emaste (mentor)
Differential Revision: https://reviews.freebsd.org/D19603
The code assumed that disks (devices) used for testing are always named
like /dev/foo, but there is no reason for that restriction and we can
easily support paths like /dev/stripe/bar.
The change is to account for a different order in which the recursive
destroy may be attempted. If we first try a dataset that can be destroyed
then it will be destroyed, but if we first try a dataset that cannot be
destroyed then we will not attempt to destroy the other dataset.
It was originally written by Sun as part of the STF (Solaris test framework).
They open sourced it in OpenSolaris, then HighCloud partially ported it to
FreeBSD, and Spectra Logic finished the port. We also added many testcases,
fixed many broken ones, and converted them all to the ATF framework. We've had
help along the way from avg, araujo, smh, and brd.
By default most of the tests are disabled. Set the disks Kyua variable to
enable them.
Submitted by: asomers, will, justing, ken, brd, avg, araujo, smh
Sponsored by: Spectra Logic Corp, HighCloud
